PDF Optimal High-Frequency Market Making

A wide variety of RL techniques have been developed to allow the agent to learn from the rewards it receives as a result of its successive interactions with the environment. A notable example is Google’s AlphaGo project , in which a deep reinforcement learning algorithm was given the rules of the game of Go, and it then taught itself to play so well that it defeated the human world champion. AlphaGo learned by playing against itself many times, registering the moves that were more likely to lead to victory in any given situation, thus gradually improving its overall strategies. The same concept has been applied to train a machine to play Atari video games competently, feeding a convolutional neural network with the pixel values of successive screen stills from the games . The goal of this paper is first to propose an optimal quoting strategy that is adopted by the stochastic volatility, drift effect and market impact by the amount and type of the orders in the price dynamics.

This consideration makes rb and ra reasonable reference prices around which to construct the market maker’s spread. Avellaneda and Stoikov define rb and ra, however, for a passive agent with no orders in the limit order book. In practice, as Avellaneda and Stoikov did in their original paper, when an agent is running and placing orders both rb and ra ra are approximated by the average of the two, r .

IEEE Transactions on Knowledge and Data Engineering

To start filling Alpha-AS https://www.beaxy.com/ and training the model (Section 5.2). Therefore, by choosing a Skew value the Alpha-AS agent can shift the output price upwards or downwards by up to 10%. Mean decrease impurity , a feature-specific measure of the mean reduction of weighted impurity over all the nodes in the tree ensemble that partition the data samples according to the values of that feature . Where the 0 subscript denotes the best orderbook price level on the ask and on the bid side, i.e., the price levels of the lowest ask and of the highest bid, respectively.

From this point, the agent can gradually diverge as it learns by operating in the changing market. We were able to achieve some parallelisation by running five backtests simultaneously on different CPU cores. Upon finalization of the five parallel backtests, the five respective memory replay buffers were merged.

What is the reservation price?

2, we set the framework in continuous time and formulate the optimization problem in terms of the expected return of the trader. Section3 is dedicated to the study of the stochastic control and Hamilton-Jacobi-Bellman equations for the model proposed in Sect. 3.2.1, we consider the case of the jumps in volatility of the price. The paper is also equipped with an Appendix on how to use the method of finite differences for the numerical solution of the corresponding nonlinear differential equation. This work presents RAGE, a novel strategy designed for solving combinatorial optimization problems where we intend to select a subset of elements from a very large set of candidates.

Gen-AS outperformed the two other baseline models on all indicators, and in turn the two Alpha-AS models substantially outperformed Gen-AS on Sharpe, Sortino and P&L-to-MAP. Localised excessive risk-taking by the Alpha-AS models, as reflected in a few heavy dropdowns, is a source of concern for which possible solutions are discussed. In most of the many applications of RL to trading, the purpose is to create or to clear an asset inventory. The more specific context of market LINK making has its own peculiarities. DRL has been used generally to determine the actions of placing bid and ask quotes directly [23–26], that is, to decide when to place a buy or sell order and at what price, without relying on the AS model. Spooner proposed a RL system in which the agent could choose from a set of 10 spread sizes on the buy and the sell side, with the asymmetric dampened P&L as the reward function (instead of the plain P&L).

Other indicators, such as the Sortino ratio, can also be used in the reward function itself. Another approach is to explore risk management policies that include discretionary rules. Alternatively, experimenting with further layers to learn such policies autonomously may ultimately yield greater benefits, as indeed may simply altering the number of layers and neurons, or the loss functions, in the current architecture. Maximum drawdown registers the largest loss of portfolio value registered between any two points of a full day of trading.

Thus, the Alpha-AS models came 1stand 2nd on 20 out of the 30 test days (67%). The mean and the median of the Sharpe ratio over all test days was better for both Alpha-AS models than for the Gen-AS model , and in turn the Gen-AS model performed significantly better on Sharpe than the two non-AS baselines. The results obtained suggest avenues to explore for further improvement. First, the reward function can be tweaked to penalise drawdowns more directly.

I. How distant is the trader’s current inventory position is from the target position? (q)

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

An amount in seconds, which is the duration for the placed limit orders. The limit bid and ask orders are canceled, and new orders are placed according to the current mid-price and spread at this interval. Trading strategy with stochastic volatility in a limit order book market. Consequently, we support our findings by comparing the models proposed within this research with the stock price impact models existing in literature. Last but not least, we have substantially improved the performances of a market maker with the proposed models.

The target for the random forest classifier is simply the sign of the difference in mid-prices at the start and the end of each 5-second timestep. That is, classification is based on whether the mid-price went up or down in each timestep. The Q-value iteration algorithm assumes that both the transition probability matrix and the reward matrix BNB are known. Hasselt, Guez and Silver developed an algorithm they called double DQN.

10 such training iterations were completed, all on data from the same full day of trading, with the memory replay buffer resulting from each iteration fed into the next. The replay buffer obtained from the final iteration was used as the initial one for the test phase. At this point the trained neural network model had 10,000 rows of experiences and was ready to be tested out-of-sample against the baseline AS models. Following the approach in López de Prado , where random forests are applied to an automatic classification task, we performed a selection from among our market features , based on a random forest classifier.

Regarding the latter, our results lead to new and easily interpreavellaneda & stoikov closed-form approximations for the optimal quotes, both in the finite-horizon case and in the asymptotic regime. Consequently, the Alpha-AS agent adapts its bid and ask order prices dynamically, reacting closely (at 5-second steps) to the changing market. This 5-second interval allows the Alpha-AS algorithm to acquire experience trading with a certain bid and ask price repeatedly under quasi-current market conditions. As we shall see in Section 4.2, the parameters for the direct Avellaneda-Stoikov model to which we compare the Alpha-AS model are fixed at a parameter tuning step once every 5 days of trading data.

• Optimal strategies for market makers have been studied by academic researchers for a very long time now, with Thomas Ho and Hans Stoll starting to write about market dealers dynamics in 1980.
• Market indicators, consisting of features describing the state of the environment.
• With the risk aversion parameter, you tell the bot how much inventory risk you want to take.
• Genetic algorithms compare the performance of a population of copies of a model, each with random variations, called mutations, in the values of the genes present in its chromosomes.
• It is observed that the thickness of the market prices is correlated with the trading intensity inversely.

The agent’s action space itself can potentially also be enriched profitably, by adding more values for the agent to choose from and making more parameters settable by the agent, beyond the two used in the present study (i.e., risk aversion and skew). In the present study we have simply chosen the finite value sets for these two parameters that we deem reasonable for modelling trading strategies of differing levels of risk. This helps to keep the models simple and shorten the training time of the neural network in order to test the idea of combining the Avellaneda-Stoikov procedure with reinforcement learning. The results obtained in this fashion encourage us to explore refinements such as models with continuous action spaces. The logic of the Alpha-AS model might also be adapted to exploit alpha signals .

• 2, we set the framework in continuous time and formulate the optimization problem in terms of the expected return of the trader.
• The replay buffer obtained from the final iteration was used as the initial one for the test phase.
• An early stopping strategy is followed on 25% of the training sets to avoid overfitting.
• There is a lot of mathematical detail on the paper explaining how they arrive at this factor by assuming exponential arrival rates.
• In this study, we implement a LOB trading strategy to enter and exit the market by processing LOB data.
• Combining a deep Q-network (see Section 4.1.7) with a convolutional neural network , Juchli achieved improved performance over previous benchmarks.

As defined above, this action consists in setting the value of the risk aversion parameter, γ, in the Avellaneda-Stoikov formula to calculate the bid and ask prices, and the skew to be applied to these. The agent will place orders at the resulting skewed bid and ask prices, once every market tick during the next 5-second time step. One of the most active areas of research in algorithmic trading is, broadly, the application of machine learning algorithms to derive trading decisions based on underlying trends in the volatile and hard to predict activity of securities markets. Machine learning approaches have been explored to obtain dynamic limit order placement strategies that attempt to adapt in real time to changing market conditions. As regards market making, the AS algorithm, or versions of it , have been used as benchmarks against which to measure the improved performance of the machine learning algorithms proposed, either working with simulated data or in backtests with real data. The literature on machine learning approaches to market making is extensive.

We plan to use such approximations in further tests with our RL approach. The performance results for the 30 days of testing of the two Alpha-AS models against the three baseline models are shown in Tables 2–5. All ratios are computed from Close P&L returns (Section 4.1.6), except P&L-to-MAP, for which the open P&L is used. Figures in bold are the best values among the five models for the corresponding test days.

A continuous action space, as the one used to choose spread values in , may possibly perform better, but the algorithm would be more complex and the training time greater. With the risk aversion parameter, you tell the bot how much inventory risk you want to take. A value close to 1 will indicate that you don’t want to take too much inventory risk, and hummingbot will “push” the reservation price more to reach the inventory target. This potential weakness of the analytical AS approach notwithstanding, we believe the theoretical optimality of its output approximations is not to be undervalued. On the contrary, we find value in using it as a starting point from which to diverge dynamically, taking into account the most recent market behaviour. With the above definition of our Alpha-AS agent and its orderbook environment, states, actions and rewards, we can now revisit the reinforcement learning model introduced in Section (4.1.2) and specify the Alpha-AS RL model.