Mining for Alpha: How AI is Rewriting the Rules of Quantitative Investing

Author: Denis Avetisyan

A new framework uses the power of large language models and evolutionary algorithms to automatically discover and refine predictive factors for building profitable investment strategies.

FactorEngine operates on the principle that robust investment strategies emerge not from static design, but from iterative refinement-beginning with the translation of conceptual ideas into executable code, then employing a co-evolutionary process where large-scale mutations proposed by language model agents are precisely tuned via Bayesian search and rapid validation, ultimately yielding a portfolio of elite factors primed for backtesting and continuous improvement-a system acknowledging that even the most promising approaches require constant adaptation to withstand the inevitable decay of market conditions.

FactorEngine is a program-level knowledge-infused factor mining framework leveraging Bayesian optimization to enhance alpha generation in quantitative finance.

Despite advances in automated factor discovery, identifying consistently profitable and interpretable predictive signals from noisy financial data remains a significant challenge. This paper introduces ‘FactorEngine: A Program-level Knowledge-Infused Factor Mining Framework for Quantitative Investment’, a novel system that evolves factors represented as executable code, leveraging large language models to bridge the gap between unstructured financial knowledge and algorithmic trading strategies. FactorEngine achieves state-of-the-art performance through a separation of logic and parameter optimization, and by incorporating a knowledge-infused bootstrapping module-demonstrating improved stability and portfolio impact across extensive backtests. Could this program-level approach unlock a new generation of robust and adaptable quantitative investment strategies?

The Erosion of Signal in Complex Markets

Conventional methods of identifying profitable investment signals, often termed alpha factors, frequently prove inadequate when confronted with the intricacies of modern financial markets. These approaches typically depend on either explicitly programmed trading rules, designed by human analysts, or the detection of simple statistical relationships within historical data. However, market dynamics are rarely linear; complex interactions between countless variables, combined with evolving investor behavior, create a landscape where superficial correlations quickly erode. Consequently, factors discovered through these traditional means often exhibit limited predictive power outside of the specific historical period used for their development, failing to adapt to changing market conditions and ultimately hindering consistent performance. The reliance on handcrafted logic or shallow analysis overlooks the subtle, non-linear relationships crucial for navigating increasingly complex financial ecosystems.

Traditional approaches to identifying predictive investment signals, known as alpha factors, frequently fall short when confronted with the intricacies of real-world financial markets. These methods, often based on simple statistical relationships, struggle to discern the subtle, non-linear connections that drive asset pricing. Consequently, factors discovered through these means demonstrate limited resilience; their predictive power tends to erode as market conditions evolve. A factor effective during periods of economic expansion may prove useless – or even detrimental – during a recession, highlighting a critical lack of adaptability. This inflexibility stems from an inability to account for shifting investor behavior, changing macroeconomic forces, and the dynamic interplay between various market participants, ultimately limiting the long-term viability of these conventionally-derived strategies.

The proliferation of financial data, stemming from high-frequency trading, alternative datasets, and increasingly interconnected markets, presents a significant challenge to traditional methods of identifying predictive investment signals – known as alpha factors. Simply put, the sheer volume and intricacy of this data exceed the capacity of rule-based or superficial statistical analyses to effectively discern meaningful patterns. This isn’t merely a matter of computational power; the relationships within modern financial data are often non-linear, time-varying, and influenced by complex interactions that defy simple observation. Consequently, a shift towards more sophisticated techniques – leveraging machine learning, artificial intelligence, and advanced statistical modeling – becomes not just desirable, but essential for uncovering robust and adaptable investment strategies capable of navigating today’s dynamic markets. The pursuit of alpha, therefore, demands an evolution beyond traditional approaches, embracing intelligence that can decipher the subtle nuances hidden within the deluge of financial information.

Across both the CSI300 and CSI500 markets, the agent-based methods demonstrate varying cumulative excess returns, and their underlying factor correlations, as visualized by multidimensional scaling (MDS), reveal distinct relationships between these strategies.

FactorEngine: A Paradigm Shift in Factor Evolution

FactorEngine utilizes a combined methodology for alpha factor discovery, moving beyond traditional methods that focus solely on statistical relationships. It integrates program-level logic evolution – where the computational structure of a factor is dynamically altered – with Bayesian hyperparameter optimization. This allows for both the structure of a factor and its internal parameters to be optimized concurrently. Bayesian optimization efficiently explores the hyperparameter space, while program evolution introduces novelty and adaptability to the factor itself. This integrated approach aims to overcome the limitations of hand-crafted factors and static rule-based systems by discovering factors with complex, dynamically adjusting logic.

Macro-Micro Co-evolution within FactorEngine separates the optimization of a factor’s computational logic – the ‘macro’ level – from the tuning of its numerical parameters – the ‘micro’ level. Traditional alpha factor research often conflates these two processes, leading to inefficient searches where parameter adjustments mask or are limited by suboptimal logic. By decoupling them, FactorEngine enables independent evolution of both aspects. The macro-level employs program synthesis techniques to explore different algorithmic structures, while the micro-level utilizes Bayesian optimization to refine parameter settings for the current logic. This parallel optimization significantly expands the search space and accelerates the discovery of high-performing factors compared to methods that treat the factor as a static, parameterized function.

FactorEngine differentiates itself from traditional alpha factor construction by representing factors as executable programs subject to evolutionary algorithms. This programmatic approach allows for the creation of dynamic strategies that respond to changing market conditions, overcoming the inflexibility inherent in static, rule-based factors. Instead of manually defining and optimizing a fixed set of rules, FactorEngine iteratively modifies the underlying program logic – the factor’s code – through techniques like genetic programming. This enables the discovery of non-linear relationships and complex interactions within data that would be difficult or impossible to identify using conventional statistical methods, ultimately leading to factors with increased robustness and adaptability.

Employing Bayesian parameter search accelerates performance gains compared to non-Bayesian tuning, and GPT-4o consistently outperforms Gemini-2.5-flash-lite as a backbone agent in evolved strategies.

Architectural Pillars: Decoupling Logic, Search, and Resources

Logic separation within the factor architecture decouples the fundamental conceptualization of a factor – its underlying rationale and intended effect – from the numerical parameters used to implement it. This design choice allows for iterative refinement of the core factor logic without necessitating adjustments to all associated parameter settings, and conversely, enables parameter optimization without impacting the foundational ideas. The benefit is increased innovation; developers can explore alternative factor implementations or modify existing ones without the risk of unintentionally altering the behavior of unrelated components, leading to faster experimentation and more robust system evolution.

Search Strategy Separation employs a dual-faceted approach to parameter optimization. Initially, Large Language Models (LLMs) are utilized for directional search, rapidly identifying promising regions within the parameter space based on high-level objectives. This broad exploration is then refined through automated Bayesian optimization, which systematically fine-tunes parameters within those identified regions to maximize performance. This hybrid strategy combines the LLM’s capacity for rapid, informed exploration with the precision of Bayesian optimization, resulting in an efficient and robust search process that surpasses the capabilities of either method operating independently.

Resource Separation within the system architecture dynamically allocates computational tasks between Large Language Models (LLMs) and locally available resources. This distribution is determined by the characteristics of each task; LLMs handle high-level reasoning and broad contextual understanding, while local resources-including CPUs and GPUs-execute computationally intensive operations such as data processing and model fine-tuning. This approach minimizes reliance on LLM inference for every step, reducing latency and cost. Furthermore, it allows for parallel execution, maximizing throughput and ensuring efficient utilization of available hardware, ultimately optimizing performance and scalability.

Enhancing Exploration and Ensuring Factor Stability

Multi-Island Evolution within FactorEngine operates by maintaining multiple, independent evolutionary populations – termed ‘islands’ – that evolve in parallel. Periodic migration of individuals between these islands introduces genetic diversity, preventing any single island from converging prematurely on a suboptimal solution. This methodology contrasts with traditional evolutionary algorithms that utilize a single population, which are prone to losing diversity and becoming trapped in local optima. The migration rate and island topology are configurable parameters, allowing optimization of the balance between exploration and exploitation during the factor discovery process. By fostering diversity across multiple populations, Multi-Island Evolution enhances the robustness and generalizability of the discovered factors.

The Chain-of-Experience (CoE) functions as a repository of past evolutionary runs, storing data on factor performance, mutation sequences, and environmental conditions encountered during prior iterations. This historical data is leveraged to inform subsequent exploration by biasing the evolutionary process towards promising areas of the factor space. Specifically, the CoE enables the transfer of knowledge from previously successful trajectories, allowing the FactorEngine to prioritize mutations and parameter adjustments that have demonstrated robustness. This accelerates the discovery of stable factors by reducing redundant exploration and focusing computational resources on refining proven concepts, ultimately improving the efficiency and reliability of the factor generation process.

Factor initialization within the FactorEngine leverages ‘Report-Inspired Seed’ programs to accelerate factor discovery and improve performance. These seed programs are not randomly generated; instead, they are derived directly from existing, peer-reviewed financial research and established investment strategies documented in industry reports. This approach provides a structured starting point for the evolutionary process, effectively biasing initial factor formulations toward potentially profitable and theoretically sound concepts. By grounding factor development in pre-existing knowledge, the engine reduces the search space and focuses computational resources on refining and optimizing proven methodologies, rather than exploring entirely novel, and potentially unproductive, approaches.

Validating Performance and Charting Future Directions

FactorEngine’s automatically generated investment strategies demonstrate compelling performance characteristics when evaluated using established financial metrics. The system’s evolved factor programs consistently produce positive results, as indicated by a robust Information Coefficient (IC) of 0.0474 within the CSI300 index and an even higher IC of 0.0536 in the CSI500. These IC values represent the correlation between predicted and actual returns, signifying the predictive power of the generated factors. Further supporting these findings, the system delivers an Excess Annual Return of 18.99% in the CSI300 market and an Annualized Return (AR) of 8.36% in the CSI500, showcasing its capacity to generate substantial value through data-driven factor discovery and refinement. The combination of strong IC and IR scores suggests a reliable and efficient investment approach, validating the efficacy of the framework’s automated strategy generation process.

Rigorous evaluation demonstrates FactorEngine’s predictive power through its achieved Information Coefficient (IC) scores. In the highly competitive CSI300 market, the system attained an IC of 0.0474, indicating a discernible ability to differentiate between positive and negative returns. This performance extends to the broader CSI500 market, where FactorEngine yielded an even stronger IC of 0.0536. These scores represent a quantifiable measure of the factors’ skill in forecasting asset returns, suggesting that the automatically discovered strategies hold substantial potential for generating alpha and outperforming benchmark indices.

Demonstrating tangible financial results, the FactorEngine framework yielded an 18.99% Excess Annual Return within the CSI300 market, indicating performance exceeding a benchmark index. Complementing this, analysis of the CSI500 market revealed an 8.36% Annualized Return $(AR)$ , signifying the system’s capability to generate consistent gains across varying market segments. These returns suggest the evolved factor programs, automatically generated by the engine, are not only statistically significant, as measured by Information Coefficient and Ratio, but also deliver substantial value when applied to real-world investment scenarios. The observed performance underscores the potential for automated factor discovery to drive superior investment outcomes and reduce dependence on traditional, labor-intensive methods.

Traditional quantitative investment strategies heavily depend on painstaking manual feature engineering, requiring experts to identify and construct predictive factors from vast datasets. However, this approach is often slow, resource-intensive, and susceptible to human bias. FactorEngine addresses these limitations by automating the factor discovery and refinement process. The framework leverages large language models to systematically search for and optimize investment factors, effectively diminishing the need for extensive manual intervention and subjective judgment. This automated capability not only accelerates the development of new strategies but also potentially uncovers non-intuitive factors that might be overlooked through conventional methods, offering a significant advancement in quantitative finance.

The continued development of FactorEngine centers on leveraging the complementary strengths of reinforcement learning and large language models. Integrating reinforcement learning algorithms promises to enable the framework to dynamically optimize factor combinations and trading strategies based on market feedback, moving beyond static factor definitions. Simultaneously, enhancing the LLM-driven search capabilities will broaden the scope of potential factors considered, allowing for the identification of more nuanced and previously overlooked relationships within financial data. This synergistic approach aims to create a self-improving system capable of adapting to evolving market conditions and consistently discovering high-performing factors with minimal human intervention, ultimately pushing the boundaries of automated investment strategy development.

The pursuit of robust investment factors, as detailed in FactorEngine, mirrors a fundamental principle of enduring systems. The framework’s iterative refinement of predictive factors through large language models and Bayesian optimization isn’t merely about discovering ‘alpha’; it’s about building resilience against the inevitable decay of predictive power. As David Hilbert observed, “We must be able to answer the question: what are the ultimate foundations of mathematics?” Similarly, FactorEngine seeks the underlying foundations of financial prediction, acknowledging that even the most promising factors require constant evolution to maintain their efficacy. The program-level approach, while complex, is a testament to the idea that architecture without a robust foundation-or, in this case, a constantly evolving factor model-is ultimately fragile.

What Lies Ahead?

The pursuit of predictive factors in quantitative finance, as exemplified by FactorEngine, feels less like discovery and more like a protracted negotiation with entropy. The system refines, evolves, and momentarily delays the inevitable regression to the mean. FactorEngine’s synthesis of large language models and evolutionary algorithms represents a sophisticated attempt to map fleeting correlations, but correlation, as always, is not causation-merely a temporary alignment of noise. The true challenge isn’t generating factors, but understanding why they decay.

Future iterations will likely focus on enhancing the robustness of these factors, attempting to shield them from the ceaseless currents of market change. However, a more fundamental question remains: can a system built on identifying patterns truly transcend the randomness inherent in complex systems? The framework’s dependence on historical data, while pragmatic, introduces a bias towards the past-a past that, by definition, no longer exists.

Perhaps the most fruitful avenue for exploration lies not in refining the detection of factors, but in modeling their lifespan. A predictive theory of factor decay-one that acknowledges time not as a variable, but as the very medium of their existence-may prove more valuable than any endlessly iterating search for alpha. Stability, after all, is often just a delayed manifestation of instability.

Original article: https://arxiv.org/pdf/2603.16365.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/