Smart Bidding: AI Masters the Closing Auction

by

Author: Denis Avetisyan

Researchers have developed a reinforcement learning system that intelligently navigates closing auctions, outperforming conventional market-making strategies.

Training analysis reveals the iterative refinement of a system, where successive evaluations progressively align performance with desired outcomes.
Training analysis reveals the iterative refinement of a system, where successive evaluations progressively align performance with desired outcomes.

This paper introduces a Deep Q-Network framework for optimizing market making in the presence of a closing auction, explicitly addressing inventory risk and order book dynamics.

Traditional optimal market-making models often neglect the liquidity events inherent in closing auctions, creating a gap in end-of-day risk management. This work, ‘Learning Market Making with Closing Auctions’, addresses this limitation by introducing a Deep Q-Learning framework designed to explicitly incorporate and anticipate closing auction dynamics. The proposed method demonstrably outperforms classical benchmarks and theoretical approaches when applied to both simulated and historical S&P 500 asset data. Could this reinforcement learning approach offer a more robust solution for navigating complex order book dynamics and maximizing profitability in modern trading environments?

The Architecture of Liquidity: Foundations of Market Making

The smooth functioning of financial markets hinges on effective market making, a process where intermediaries provide liquidity by simultaneously posting buy and sell orders for an asset. This continuous presence narrows the bid-ask spread – the difference between the highest buy order and the lowest sell order – directly reducing transaction costs for investors. However, achieving this efficiency is far from simple. Market makers face inherent challenges stemming from the unpredictable nature of order flow, the risk of holding unsold inventory (particularly during market downturns), and the ever-present threat of adverse selection – attracting more informed traders who can exploit price discrepancies. Successfully balancing these competing forces demands constant adaptation and increasingly sophisticated strategies to maintain profitability while fulfilling the vital role of facilitating seamless trading.

Conventional market-making approaches often falter when confronted with the complex relationships between incoming orders, the risk of holding unsold assets (inventory risk), and the challenge of adverse selection. This arises because informed traders possess private information, leading them to trade selectively, potentially leaving market makers with losing positions. Traditional strategies, relying on static or slowly-adjusting quotes, struggle to differentiate between genuine demand and opportunistic trading by these informed participants. Consequently, market makers face a continuous balancing act: widening bid-ask spreads to protect against adverse selection increases transaction costs and reduces liquidity, while maintaining narrow spreads exposes them to greater risk. This dynamic interplay demands constant adaptation and increasingly sophisticated algorithms to effectively manage these competing forces and provide consistent, efficient price discovery.

Effective market making hinges on a nuanced approach to price quotation and inventory control, demanding strategies that transcend simple buy-sell orders. Sophisticated algorithms continuously assess incoming order flow, predicting short-term price movements and adjusting bid-ask spreads to attract trades while mitigating risk. Crucially, these systems must account for adverse selection – the tendency for informed traders to exploit less informed market makers – by subtly shifting prices to discourage unfavorable trades. Furthermore, managing inventory is paramount; holding too much of an asset exposes the market maker to losses if the price declines, while insufficient inventory limits the ability to fulfill buy orders. Consequently, advanced strategies employ statistical modeling and machine learning to dynamically adjust inventory levels, balancing the cost of holding assets against the risk of missing profitable trading opportunities and maintaining a consistently stable market.

Adaptive Intelligence: Reinforcement Learning for Optimal Execution

Reinforcement learning (RL) provides a framework for market making policy optimization through iterative learning from simulated or live market interactions. Unlike traditional methods relying on pre-defined rules or static models, RL algorithms learn by receiving rewards or penalties for their actions – in this case, order placement and adjustment – and adapting their strategies to maximize cumulative reward, typically representing profit. This learning process allows the agent to discover optimal policies without explicit programming for specific market scenarios, enabling adaptation to complex, non-linear market dynamics and potentially outperforming rule-based systems. The agent’s policy is refined through trial and error, balancing the exploration of new strategies with the exploitation of known profitable actions.

Reinforcement learning algorithms demonstrate adaptability in dynamic market environments by continuously refining trading strategies based on observed outcomes. This is particularly beneficial in closing auctions, where price discovery occurs rapidly and unpredictably. Unlike static strategies, RL agents can learn to adjust bid and ask prices, order sizes, and timing in response to shifts in order flow, volatility, and participant behavior. The algorithms optimize for reward functions, typically designed to maximize profitability while minimizing risk, leading to improved execution quality and potentially higher returns compared to rule-based or static approaches. This learning process allows the algorithm to identify and exploit subtle market inefficiencies that would be difficult to capture with pre-defined rules.

Effective implementation of reinforcement learning (RL) for optimal execution necessitates a strong foundation in stochastic control theory due to the inherent uncertainty present in financial markets. Stochastic control provides the mathematical tools to model and manage these uncertainties, allowing RL agents to make informed decisions under conditions where future outcomes are not fully predictable. Specifically, concepts like Markov Decision Processes (MDPs), Bellman equations, and dynamic programming are crucial for formulating the trading problem as a stochastic control problem, enabling the agent to learn an optimal policy that maximizes cumulative rewards – typically profit – while accounting for market volatility and order book dynamics. The agent must estimate the probability distribution of future states and select actions that optimize expected returns, a process directly addressed by stochastic control techniques.

A Neural Framework: The NFQ Algorithm in Action

The Neural-Fitted Q-learning (NFQ) algorithm represents a deep reinforcement learning approach to automated market making. Unlike traditional methods relying on predefined rules or theoretical models, NFQ employs a deep neural network to approximate the optimal Q-function, which estimates the expected cumulative reward for taking a specific action in a given market state. This allows the agent to learn directly from market data and adapt its trading strategy in response to changing conditions. The algorithm is designed to manage inventory and execute orders with the goal of maximizing profit, and is applicable to continuous limit order book (CLOB) environments. By utilizing deep learning, NFQ can effectively handle the complexities and high dimensionality inherent in modern financial markets, offering a data-driven alternative to analytical market making strategies.

The Neural-Fitted Q-learning (NFQ) algorithm addresses the challenges of applying reinforcement learning to market making by employing deep neural networks as function approximators for the Q-function. Traditional Q-learning methods struggle with the high dimensionality inherent in order book data and the continuous state and action spaces of financial markets. NFQ overcomes these limitations by mapping high-dimensional state representations – encompassing order book imbalances, price levels, and inventory positions – to Q-values, which estimate the expected cumulative reward for taking specific actions. This neural network approximation allows the agent to generalize learned knowledge across similar states, effectively handling the complexity of real-world trading environments and enabling decision-making in scenarios with a vast number of possible states and actions.

The Neural-Fitted Q-learning (NFQ) algorithm optimizes inventory management and execution strategies by leveraging a deep neural network to approximate the optimal Q-function. This allows the agent to dynamically adjust bid and ask prices, and order sizes, based on the current state of the continuous limit order book (CLOB). By learning an optimal policy, NFQ minimizes adverse selection costs and maximizes profitability through efficient inventory control. Empirical results demonstrate that this data-driven approach consistently surpasses the performance of traditional market-making strategies, such as the Avellaneda-Stoikov (AS) model and the Time-Weighted Average Price (TWAP) strategy, in both simulation and historical market data.

Empirical results detailed in this paper demonstrate the superior performance of the Neural-Fitted Q-learning (NFQ) agent when benchmarked against both the Avellaneda-Stoikov (AS) theoretical model and the Time-Weighted Average Price (TWAP) strategy. These findings are consistent across both simulated market environments and analyses utilizing historical order book data. Specifically, the NFQ agent consistently achieves higher cumulative returns than both comparative strategies.

Performance analysis of the Neural-Fitted Q-learning (NFQ) agent reveals substantial reward accumulation during the continuous limit order book (CLOB) phase of training, indicating consistent learning and avoidance of divergence. This stability is a key characteristic contributing to the algorithm’s improved performance and reliability in dynamic market environments, suggesting robust convergence properties during operation.

Empirical results demonstrate that the Neural-Fitted Q-learning (NFQ) algorithm consistently achieves higher mean returns compared to both the Avellaneda-Stoikov (AS) theoretical benchmark and the Time-Weighted Average Price (TWAP) strategy across tested market conditions. Critically, during the continuous limit order book (CLOB) phase of training, the NFQ agent exhibits stable training loss, indicating consistent learning and avoidance of divergence. This stability is a key characteristic contributing to the algorithm’s improved performance and reliability in dynamic market environments, suggesting robust convergence properties during operation.

Performance analysis of the Neural-Fitted Q-learning (NFQ) agent reveals substantial reward accumulation during the closing auction phase of trading simulations. This reward contribution is a key factor in the NFQ agent’s overall improved performance compared to both the Avellaneda-Stoikov (AS) benchmark and the Time-Weighted Average Price (TWAP) strategy. Specifically, the agent’s learned policy enables it to capitalize on price movements and order flow dynamics unique to the closing auction, generating returns not consistently achieved by the comparison strategies. Quantitative results demonstrate that gains during this phase significantly contribute to the higher mean returns observed with the NFQ agent in both simulated and historically-driven continuous limit order book (CLOB) environments.

The Evolving Landscape: Impact and Future Directions

The application of reinforcement learning to automated market making represents a significant advancement in algorithmic trading strategies. These algorithms learn to optimize quote placements and order sizes through continuous interaction with the market, effectively minimizing the price impact of their trades. Unlike traditional market making approaches that rely on pre-defined rules, reinforcement learning allows algorithms to adapt to changing market conditions and learn complex trading patterns. This dynamic adjustment results in improved order execution for clients and a more stable, efficient market overall, as algorithms prioritize minimizing adverse selection and maximizing fill rates. The ability to learn from experience, rather than relying on static models, positions reinforcement learning as a powerful tool for navigating the inherent complexities and uncertainties of financial markets.

Modern market making algorithms actively contribute to market stability and efficiency through continuous, dynamic adjustments to quoted prices and meticulous inventory management. These systems don’t simply post and wait; they constantly reassess order book dynamics, predicting short-term price movements and adapting bids and asks accordingly. This responsiveness minimizes adverse selection – the risk of trading with more informed parties – and narrows the bid-ask spread, reducing transaction costs for all participants. Furthermore, intelligent inventory control prevents the accumulation of excessive risk, as the algorithm proactively hedges positions and avoids exacerbating price volatility. The result is a more liquid, resilient market where orders are executed swiftly and at favorable prices, fostering greater confidence among traders and investors.

Ongoing investigation centers on refining these reinforcement learning-driven market making algorithms to navigate increasingly intricate market dynamics, such as those arising from order book fragmentation and adverse selection. Researchers are actively exploring the integration of more sophisticated deep learning architectures – including transformers and attention mechanisms – to enhance the algorithms’ capacity for predictive modeling and real-time adaptation. This involves not only improving the accuracy of price impact estimations but also developing strategies to effectively manage risk in volatile conditions. The ultimate goal is to create algorithms capable of learning and responding to subtle market signals, thereby minimizing transaction costs and maximizing liquidity, even amidst complex and unpredictable trading scenarios.

The convergence of reinforcement learning and sophisticated algorithms heralds a transformative shift in financial markets, promising a future where automated systems drive increasingly efficient and stable trading environments. This new era of data-driven market making moves beyond traditional, rule-based approaches, enabling algorithms to dynamically adapt to evolving market conditions and optimize order execution with unprecedented precision. The benefits of this transition extend to all market participants; traders experience reduced transaction costs and minimized market impact, while investors gain access to more liquid and resilient markets. Ultimately, these advancements are poised to unlock substantial gains in market efficiency and contribute to a more robust and accessible financial ecosystem for both institutional and individual actors.

The pursuit of optimal market making, as detailed in this work, reveals a complex interplay between immediate gains and long-term systemic health. The framework’s success hinges on anticipating the closing auction’s influence, acknowledging that each optimization introduces new, often subtle, tensions within the order book dynamics. This echoes John Locke’s assertion: “All mankind… being all equal and independent, no one ought to harm another in his life, health, liberty, or possessions.” Just as Locke emphasizes the protection of fundamental rights, this research seeks to establish a market-making strategy that safeguards against inventory risk and ensures stable, equitable exchange-a system where the ‘health’ of the market, like individual liberty, is paramount and requires constant, considered maintenance. The system’s behavior over time, not simply maximizing short-term profit, dictates its ultimate resilience.

Future Directions

The integration of reinforcement learning with limit order book dynamics, as demonstrated, offers a potent, yet predictably incomplete, solution. While the framework adeptly incorporates the complexities of a closing auction – a necessary refinement over purely continuous-time models – it simultaneously reveals the inherent fragility of such optimization. The agent learns to navigate a specific instantiation of market microstructure; generalization to markedly different auction designs or order types remains an open question, and likely requires a more robust, theoretically grounded approach to state-space representation.

A critical simplification lies in the agent’s limited awareness of systemic risk. The focus on inventory management, while pragmatic, sidesteps the larger issue of interconnectedness. A truly intelligent market maker must, at some level, model the behavior of other agents, anticipating cascading effects and potential manipulation. This demands a move beyond single-agent reinforcement learning, towards game-theoretic formulations that acknowledge the strategic interplay within the order book.

Ultimately, the pursuit of optimal market making strategies reveals a familiar truth: every clever trick has risks. The gains achieved through sophisticated algorithms are perpetually offset by the potential for unforeseen consequences. The next step isn’t simply to build more complex agents, but to develop tools for understanding – and accepting – the inherent limits of control within complex adaptive systems.

Original article: https://arxiv.org/pdf/2601.17247.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-01-27 12:01