Decoding the Market with Deep Learning

by

Author: Denis Avetisyan

A new framework combines the power of artificial intelligence with analyst insights to predict stock returns and offer economic transparency.

Return predictability significantly improves with the introduction of consensus learning, peaking at a $λ$ value between 0.3 and 0.4, and consistently outperforms benchmarks even at higher $λ$ values, while the accuracy of consensus variable approximation increases steadily with increasing $λ$.
Return predictability significantly improves with the introduction of consensus learning, peaking at a $λ$ value between 0.3 and 0.4, and consistently outperforms benchmarks even at higher $λ$ values, while the accuracy of consensus variable approximation increases steadily with increasing $λ$.

This paper introduces the Consensus-Bottleneck Asset Pricing Model, an interpretable deep learning approach leveraging macroeconomic embeddings and analyst consensus forecasts for improved return prediction.

Despite advances in asset pricing, understanding why certain stocks outperform remains a central challenge. This paper introduces the Interpretable Deep Learning for Stock Returns: A Consensus-Bottleneck Asset Pricing Model (CB-APM), a novel framework that links dispersed investor beliefs-captured via analyst consensus-to expected stock returns. By modeling a “consensus bottleneck” within a neural network, CB-APM not only improves long-horizon return prediction but also provides a structurally interpretable connection between information aggregation and asset prices. Does this approach unlock a deeper understanding of belief-driven market dynamics beyond traditional factor models?

The Fragility of Conventional Valuation

Asset pricing, as traditionally conceived, frequently stumbles when confronted with the volatile realities of investor psychology and broader economic forces. Models built on strict mathematical foundations and historical data often assume rational actors and stable conditions, failing to account for the pervasive influence of fear, greed, and herd behavior. Consequently, predictions generated by these models can diverge significantly from actual market outcomes, particularly during periods of heightened uncertainty or rapid change. The limitations stem from an inability to fully integrate the complex interplay between macroeconomic indicators-such as inflation, interest rates, and employment-and the often-irrational exuberance or pessimism that drives collective investment decisions. This disconnect underscores the need for more sophisticated approaches that acknowledge the behavioral and contextual factors shaping asset valuations, moving beyond purely quantitative analysis to embrace a more holistic understanding of market dynamics.

Asset pricing models heavily dependent on past data encounter significant limitations when confronted with the ever-changing landscape of financial markets. These models often assume that historical relationships between variables will persist, a premise quickly invalidated by structural breaks and novel events. The emergence of previously unseen technologies, shifts in global economic power, or unexpected geopolitical crises introduce dynamics that are absent from historical datasets. Consequently, relying solely on past performance can lead to substantial mispricing of assets and inaccurate forecasts, as the models fail to adapt to fundamentally altered conditions. The increasing frequency of ‘black swan’ events – unpredictable occurrences with extreme impact – further underscores the inadequacy of purely historical approaches, highlighting the necessity of incorporating forward-looking indicators and more flexible modeling techniques.

Conventional asset pricing models often operate with a distinctly quantitative focus, overlooking the substantial influence of subjective assessments and expert opinions. Research indicates that incorporating qualitative data, such as analyst consensus ratings or sentiment extracted from news articles, can significantly improve predictive accuracy regarding asset returns. The omission of these “soft” factors represents a critical limitation, as collective expert judgment often distills complex, non-numerical information-like a company’s competitive advantages or potential disruptions-that purely statistical models miss. Consequently, forecasts generated by traditional models can be systematically biased, failing to capture the full spectrum of influences driving market behavior and ultimately hindering investment strategies reliant on accurate return predictions. This highlights a growing need for hybrid approaches that effectively integrate both quantitative rigor and qualitative insights to better understand and anticipate financial market dynamics.

Without incorporating macroeconomic state embeddings, the model achieves monthly R2R² for annual stock return estimation and average R2R² for approximating analysts’ consensus variables.
Without incorporating macroeconomic state embeddings, the model achieves monthly R2R² for annual stock return estimation and average R2R² for approximating analysts’ consensus variables.

A Consensus-Driven Approach to Asset Pricing

The Consensus-Bottleneck Asset Pricing Model (CB-APM) functions as a dual-purpose predictive framework, aiming to both replicate the collective judgments of financial analysts and forecast future asset returns. This is achieved by training the model to minimize the error between its predicted returns and both historical asset returns and the consensus forecasts generated by financial analysts covering those assets. By simultaneously targeting these two objectives, CB-APM seeks to identify factors that are not only correlated with future returns but also align with the information considered relevant by professional analysts, potentially offering improved predictive accuracy and interpretability compared to traditional asset pricing models.

The Concept Bottleneck architecture within the CB-APM enforces a specific learning process by requiring the model to first generate a set of interpretable, high-level concepts from input data. This is achieved through an intermediate layer, termed the ‘concept layer’, which is constrained to a relatively low dimensionality. By forcing the model to compress information into these concepts before making predictions about asset returns, the architecture promotes interpretability and allows for analysis of the learned concepts themselves. This contrasts with traditional ‘end-to-end’ models, where the model directly maps inputs to outputs without explicit concept formation, and facilitates understanding why the model makes certain predictions.

The Consensus-Bottleneck Asset Pricing Model (CB-APM) employs an autoencoder to generate ‘Macroeconomic State Embeddings’ from a high-dimensional macroeconomic data set. This process reduces the original data’s dimensionality while preserving key contextual information relevant to asset pricing. The autoencoder is trained to reconstruct the input data from a compressed, lower-dimensional representation, effectively learning a non-linear mapping that distills the most salient macroeconomic factors. The resulting Macroeconomic State Embeddings serve as a concise input for subsequent layers of the CB-APM, reducing computational complexity and potentially mitigating the effects of noise present in the original high-dimensional data. This embedding technique allows the model to focus on essential macroeconomic conditions rather than being overwhelmed by granular data points.

Consensus-based active portfolio management (CB-APM) models, leveraging return forecasts and monthly rebalancing, significantly outperform both a naive neural network and the S&P 500 buy-and-hold benchmark, demonstrating the benefits of consensus learning for investment strategies.
Consensus-based active portfolio management (CB-APM) models, leveraging return forecasts and monthly rebalancing, significantly outperform both a naive neural network and the S&P 500 buy-and-hold benchmark, demonstrating the benefits of consensus learning for investment strategies.

Validating Predictive Performance

The predictive power of the model was quantitatively assessed using the $R^2$ coefficient, which measures the proportion of variance in both investment returns and consensus analyst estimates explained by the model. Out-of-sample evaluation yielded a peak $R^2$ of 10.46% achieved when employing a regularization parameter of 0.3. This indicates the model accounts for approximately 10.46% of the total variance observed in the dependent variables during the holdout period, providing a measure of its explanatory capability and generalization performance.

To maintain data integrity and model stability in the presence of incomplete firm characteristic data, the ‘Last Observation Carried Forward’ (LOCF) method was implemented. This imputation technique substitutes missing values with the most recently available data point for each firm. LOCF avoids data loss due to missingness, allowing for the inclusion of a larger sample size in the analysis and preventing potential biases introduced by other imputation methods or listwise deletion. The application of LOCF contributes to the robustness of the model by minimizing the impact of sporadic data gaps on predictive accuracy and ensuring a comprehensive dataset for evaluating model performance.

Comparative analysis was conducted to assess the predictive accuracy of CB-APM against a Linear Regression benchmark. Statistical testing demonstrated a significant improvement in CB-APM’s performance; specifically, the model consistently outperformed Linear Regression across validation datasets. The observed improvement is not attributable to chance, as determined by established statistical methods, indicating CB-APM’s superior ability to model and predict the target variable compared to the benchmark. The magnitude of the improvement was quantified through various metrics, confirming the robustness of the findings.

Analysis of the CB-APM model incorporated transaction costs to assess practical profitability. Simulations were conducted to determine model performance after accounting for trading expenses, ranging up to 75 basis points. Results indicate that the model sustains Sharpe ratios exceeding one, even under these cost conditions. This suggests the model can generate positive risk-adjusted returns despite real-world trading frictions and maintains viability for implementation in a live trading environment. The evaluation utilized actual trading costs to provide a realistic assessment of net profitability.

Expanding window evaluation reveals that the best-performing model (λ=0.65) consistently achieves higher out-of-sample R2R² scores for quarterly stock returns compared to a naive neural network (λ=0).
Expanding window evaluation reveals that the best-performing model (λ=0.65) consistently achieves higher out-of-sample R2R² scores for quarterly stock returns compared to a naive neural network (λ=0).

Portfolio Implications and Broader Significance

A rigorous evaluation of the CB-APM model involved the implementation of a ‘Long-Short Portfolio’ strategy, designed to capitalize on predicted asset price movements. Results demonstrated a significant capacity to generate alpha – exceeding the returns of standard market benchmarks – indicating the model’s predictive power in real-world investment scenarios. This strategy actively constructed portfolios by taking long positions in assets forecasted to appreciate and short positions in those expected to decline, effectively leveraging the model’s insights to navigate market fluctuations and enhance portfolio performance. The observed outperformance suggests CB-APM offers a valuable tool for investors seeking to improve risk-adjusted returns and potentially unlock superior investment outcomes.

Assessment of the model’s efficacy utilized the Sharpe Ratio, a metric quantifying risk-adjusted return, and demonstrated a compelling capacity for generating superior investment performance. Results indicated the model achieved a peak Sharpe Ratio of 1.44, signifying that for every unit of risk assumed, the model generated $1.44 in excess return – a substantial improvement over many traditional investment strategies. This figure suggests the model not only identifies potentially profitable assets but also does so while maintaining a favorable balance between risk and reward, offering investors a potentially more efficient path to achieving their financial goals. The consistently high Sharpe Ratio reinforces the model’s robustness and its capacity to navigate varying market conditions while delivering compelling, risk-adjusted returns.

The CB-APM model demonstrates a notable capacity for discerning the influence of National Bureau of Economic Research (NBER) designated recessionary periods on asset valuations. Analysis reveals the model doesn’t simply register these events, but actively interprets their impact, allowing for a more nuanced understanding of how macroeconomic shifts affect pricing dynamics. This capability extends beyond mere correlation; the model identifies specific asset classes most vulnerable during downturns and anticipates subsequent recovery patterns with increased accuracy. By effectively incorporating recessionary signals, the CB-APM provides investors with a proactive tool for portfolio adjustment, potentially mitigating losses and capitalizing on opportunities presented by evolving economic conditions, ultimately contributing to a more informed and resilient investment strategy.

The CB-APM model distinguishes itself through a synthesis of both qualitative and quantitative data in asset pricing, resulting in a demonstrably more robust and holistic evaluation than traditional methods. This integrated approach allows for a nuanced understanding of market signals, moving beyond purely numerical analysis to incorporate contextual factors often overlooked. Empirical evidence supports this claim, as the model achieves an adjusted $R^2$ of 8.35% when utilizing CB-APM consensus-a significant improvement over the 0.40% attained through raw analyst consensus. This enhanced predictive power has direct implications for investment strategies, potentially leading to more informed portfolio construction, and for risk management, allowing for a more accurate assessment of potential market vulnerabilities and improved mitigation strategies.

A two-dimensional projection of the autoencoder's latent state reveals a cyclical pattern correlated with NBER recession periods (marked by dashed ovals in 2001 and 2007-2009), with monthly variations visible in the colored data points.
A two-dimensional projection of the autoencoder’s latent state reveals a cyclical pattern correlated with NBER recession periods (marked by dashed ovals in 2001 and 2007-2009), with monthly variations visible in the colored data points.

The pursuit of predictive accuracy often leads to models of immense complexity, obscuring the underlying logic. This work, however, prioritizes clarity through the Consensus-Bottleneck Asset Pricing Model, distilling information into economically meaningful concepts. It echoes Carl Sagan’s sentiment: “Somewhere, something incredible is waiting to be known.” The CB-APM isn’t merely a ‘black box’ forecasting tool; it actively seeks to reveal the ‘something incredible’ – the core macroeconomic factors driving stock returns – offering a transparent pathway from data to insight. The model’s emphasis on analyst consensus as a ‘bottleneck’ is a deliberate attempt to reduce noise and highlight shared understanding, mirroring a preference for elegant simplicity over convoluted detail.

What Lies Ahead?

The pursuit of predictive accuracy, particularly in a domain as noisy as asset pricing, often resembles adding layers to a palimpsest. This work, by anchoring deep learning to analyst consensus, represents a tentative excavation – a revealing of underlying structure. However, the true test isn’t merely improved correlation, but a parsimonious explanation of why. The model’s success hinges on the quality of the initial consensus data; future iterations must address the inherent biases and limitations embedded within these collective forecasts.

A critical next step involves disentangling the learned macroeconomic embeddings. Currently, these representations function as opaque vectors; transforming them into economically meaningful concepts – inflation expectations, risk aversion proxies – would move the field closer to genuine understanding. The concept bottleneck, while promising, remains a heuristic; a rigorous framework for defining and evaluating ‘concept’ quality is essential.

Ultimately, the value of interpretable AI in finance isn’t about replacing human analysts, but augmenting their abilities. The challenge lies in designing models that distill complex data into actionable insights, rather than generating another layer of abstraction. The most fruitful path forward may not be deeper networks, but more deliberate deletions – a relentless pursuit of elegant simplicity.

Original article: https://arxiv.org/pdf/2512.16251.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-12-19 15:38