Unlocking Insights from Event Data: A New Approach to Feature Discovery

Author: Denis Avetisyan

Researchers have developed a framework to automatically identify meaningful features within sequences of events, improving both performance and our understanding of the underlying data.

Embedding-Aware Feature Discovery systematically dissects sequential event data by iteratively proposing human-interpretable features, assessing their alignment with learned embeddings-regions representing encoded information-and refining both feature generation and the underlying encoder to maximize predictive power and address inherent blind spots, ultimately achieving improvements in downstream task performance and enabling targeted modifications for properties like robustness and privacy.

This paper introduces Embedding-Aware Feature Discovery (EAFD), a method that jointly reasons over learned embeddings and structured features in event sequences.

Despite advances in representation learning, production systems processing temporal event sequences-such as financial transactions-continue to heavily rely on interpretable, hand-crafted features due to latency and robustness requirements. This creates a disconnect between learned embeddings and traditional feature engineering pipelines, a challenge addressed by ‘Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences’. We introduce a unified framework, EAFD, that iteratively discovers and refines features by reasoning over both pretrained embeddings and raw event data, leveraging alignment with existing representations and identifying complementary predictive signals. Across multiple benchmarks, EAFD consistently outperforms both embedding-only and feature-based baselines, achieving state-of-the-art performance-but can these automated discovery methods further unlock the full potential of event sequence data?

Unveiling the Ghosts in the Machine: The Limits of Sequential Representation

Traditional methods of converting sequences of events into numerical representations, known as embeddings, frequently fall short when dealing with the intricate details of time-dependent data. These techniques often treat events as isolated instances, neglecting the crucial relationships defined by their order and timing. Consequently, valuable information about the dynamics of the sequence is lost during the embedding process. This simplification hinders the ability of downstream models to accurately predict future events or understand the underlying processes generating the data; the resulting embeddings act as a compressed, and often distorted, representation of the original temporal information. While useful for initial analysis, this inherent limitation necessitates more sophisticated approaches capable of preserving the nuanced characteristics of sequential data to unlock its full predictive potential.

The process of converting sequential data into fixed-length vector representations, while computationally efficient, introduces inherent limitations that researchers now characterize as ‘embedding blind spots’. These blind spots represent areas within the original data where critical predictive signals are lost during the compression process, effectively diminishing the model’s ability to accurately forecast future events. Even sophisticated techniques struggle to fully preserve the nuanced relationships present in temporal sequences, leading to a failure to capture subtle but important patterns. Consequently, predictive performance is hindered, as the model operates with an incomplete understanding of the underlying dynamics, and improvements are incremental-current methods demonstrate only marginal gains over established approaches on available datasets.

Despite ongoing refinements to embedding techniques, such as CoLES and NTP, representing the intricacies of temporal event sequences continues to pose a substantial hurdle in predictive modeling. While these methods aim to distill complex event histories into manageable vector representations, current performance gains remain modest. Recent evaluations on open-source datasets reveal a relative improvement of only 5.8% over established state-of-the-art approaches, suggesting a persistent gap in the ability of embeddings to fully capture nuanced patterns and predictive signals hidden within sequential data. This limitation indicates that a significant degree of information loss occurs during the embedding process, hindering the development of truly accurate and robust predictive models for time-dependent events.

Analysis of the Rosbank dataset reveals systematic representational blind spots across CoLES, NTP, and LLM4ES embeddings, as indicated by their <span class="katex-eq" data-katex-display="false">R^2</span> coefficients of determination. — Analysis of the Rosbank dataset reveals systematic representational blind spots across CoLES, NTP, and LLM4ES embeddings, as indicated by their $R^2$ coefficients of determination.

EAFD: Iterative Feature Discovery – A Systematic Approach to Augmentation

The Embedding-Aware Feature Discovery (EAFD) framework employs an iterative methodology to identify features that enhance the predictive power of pre-existing embeddings. This process begins with an initial set of embeddings and proceeds through repeated cycles of feature generation, evaluation, and refinement. Each iteration involves proposing new features, assessing their contribution to model performance using a defined metric, and then utilizing the results of this assessment to guide the generation of subsequent feature candidates. This cyclical approach allows EAFD to progressively discover features that are complementary to the existing embedding space, improving overall model accuracy and robustness without requiring retraining of the core embedding model itself. The iterative nature also facilitates the identification of features that address specific limitations or biases present in the original embedding.

The Embedding-Aware Feature Discovery (EAFD) framework utilizes large language models, specifically GPT-OSS, to automatically generate candidate features for model improvement. This process prioritizes two key criteria: predictive performance and human interpretability. GPT-OSS is prompted to create features that, when combined with existing embeddings, enhance model accuracy on a given task. Simultaneously, the generated features are designed to be readily understandable by human analysts, facilitating debugging and trust in the model’s decision-making process. The selection of GPT-OSS is based on its capacity for both complex text generation and controllable output, allowing for a balance between feature expressiveness and clarity.

Reflective feedback within the EAFD framework operates by analyzing the performance and characteristics of features generated in each iterative round. This analysis identifies which feature aspects contribute positively or negatively to model performance, as well as patterns in feature interpretability. These insights are then formalized as prompts and constraints that guide the large language model in subsequent feature generation cycles. Specifically, successful feature characteristics are reinforced, while unsuccessful patterns are suppressed, leading to a refinement of the feature space and an increased probability of discovering high-performing, interpretable features with each iteration. This cyclical process of generation, analysis, and refinement is central to the EAFD methodology.

EAFD effectively adapts its feature selection, distributing category proportions differently across age, gender, and regression tasks.

Quantifying Predictive Value: Performance and Interpretability Regimes

The Explainable AI Feature Discovery (EAFD) process utilizes a ‘performance regime’ focused on identifying features that directly contribute to improved accuracy in downstream tasks. This is quantitatively assessed through the ‘downstream utility score’, which measures the performance gain achieved by incorporating the discovered features into a predictive model. The performance regime prioritizes features demonstrating a statistically significant and positive correlation with the downstream utility score, effectively filtering for those with demonstrable predictive power. Features failing to meet pre-defined thresholds within the performance regime are excluded from further consideration, ensuring a focus on impactful contributions to model accuracy.

The EAFD methodology includes an interpretability regime to validate discovered features beyond performance metrics. This regime assesses whether the identified features are understandable within the context of domain expertise and existing knowledge. Features are evaluated based on their logical consistency and alignment with established understandings of the data, preventing the selection of features that, while statistically predictive, lack practical meaning or could lead to incorrect inferences. This ensures that the resulting model is not only accurate but also transparent and actionable for stakeholders.

The EAFD methodology utilizes an ‘alignment score’ to assess the correlation between predictions generated from the newly discovered features and those derived from existing embedding models; a higher alignment score indicates greater consistency and validates the utility of the identified features. Empirical results demonstrate that EAFD achieves up to a 19% relative gain in performance when applied to open-source transaction datasets. Furthermore, evaluation on a proprietary multi-target financial dataset yielded a 12.55% reduction in error rates, confirming the practical benefit of this feature discovery approach.

The area under the receiver operating characteristic curve (ROC-AUC) improves and feature composition stabilizes with each iteration of the Explainable AI Feature Decomposition (EAFD) process during Rosbank validation.

Expanding the Toolkit: LLM-Powered Feature Engineering – A Synthesis of Automation and Insight

The Enhanced Automated Feature Engineering and Discovery (EAFD) framework distinguishes itself through seamless compatibility with established feature engineering tools. Unlike isolated systems, EAFD readily integrates with methods such as CAAFE, OpenFE, and Featuretools, allowing data scientists to leverage existing expertise and infrastructure. This interoperability significantly broadens the framework’s applicability across diverse datasets and machine learning tasks. By functioning not as a replacement, but as an augmentation to current practices, EAFD enables a more flexible and powerful feature engineering pipeline, ultimately improving model performance and reducing the time required for feature creation.

The innovative framework leverages the power of Large Language Models (LLMs), specifically LLM4ES and LLMFE, to unlock hidden potential within tabular datasets. Unlike traditional feature engineering techniques that rely on pre-defined rules or statistical methods, these LLMs can understand the semantic meaning of data, generating complex and informative features that capture nuanced relationships. This capability extends beyond simple transformations; the LLMs can synthesize new features by combining existing ones in intelligent ways, effectively creating higher-order representations of the original data. The result is a significant expansion of the feature space, allowing machine learning models to discern patterns that might otherwise remain obscured, and ultimately improving predictive performance on challenging tasks.

Traditional automated machine learning (AutoML) systems often struggle with identifying the most relevant features from complex tabular data, leading to suboptimal model performance. Recent advancements, however, introduce a more discerning approach to feature discovery, moving beyond brute-force methods to prioritize potentially impactful transformations. This targeted strategy, exemplified by refinements to the CoLES framework through the integration of EAFD, demonstrates measurable improvements in predictive accuracy; specifically, a 1.20% gain was observed in churn prediction tasks. This indicates a shift towards AutoML systems capable of not simply generating features, but intelligently selecting and refining them for enhanced model outcomes, promising more robust and reliable machine learning solutions.

Removing <span class="katex-eq" data-katex-display="false">\text{mcc}</span> and <span class="katex-eq" data-katex-display="false">\text{trx_amount}</span> feature categories from embeddings demonstrates a measurable impact on <span class="katex-eq" data-katex-display="false">R2</span> reconstruction performance, as indicated by the changes in <span class="katex-eq" data-katex-display="false">\Delta R^{2}</span>. — Removing $\text{mcc}$ and $\text{trx_amount}$ feature categories from embeddings demonstrates a measurable impact on $R2$ reconstruction performance, as indicated by the changes in $\Delta R^{2}$ .

Preserving Privacy and Expanding Applications: The Ethical Imperative of Data Science

Integrating techniques like Hilbert-Schmidt Independence Criterion (HSIC) regularization with Efficient Adversarial Feature Discovery (EAFD) represents a significant step towards safeguarding sensitive data while still leveraging its analytical potential. HSIC regularization effectively limits the statistical dependence between discovered features and the original sensitive attributes, thereby minimizing the risk of data leakage during model training and deployment. This approach doesn’t simply mask data; it actively erases potentially identifying information embedded within the features themselves, offering a robust privacy guarantee. By controlling the information flow, the combined EAFD-HSIC framework allows for the creation of models that can perform complex tasks – such as fraud detection in financial transactions or disease diagnosis in healthcare – without inadvertently revealing private patient or customer details. The result is a powerful tool for responsible data science, enabling innovation while upholding ethical considerations and regulatory compliance.

The convergence of privacy-preserving feature erasure and enhanced feature attribution discovery opens significant avenues for application in highly sensitive sectors. Specifically, financial transactions and healthcare stand to benefit immensely; these domains routinely handle personally identifiable information and are subject to stringent data protection regulations. By enabling the selective removal of potentially revealing features from datasets while maintaining analytical utility, this technology facilitates secure data sharing and collaborative research. This is crucial for fraud detection in finance, where patterns must be identified without compromising individual account details, and in healthcare, where predictive modeling for disease outbreaks or personalized treatment plans requires responsible data handling. The ability to confidently deploy machine learning models without risking privacy breaches fosters trust and innovation within these critical industries.

Ongoing research prioritizes streamlining the iterative feature attribution and erasure process, aiming for increased efficiency and reduced computational cost. Investigations are also underway to evaluate the potential of novel Large Language Model (LLM) architectures – including those beyond the transformer paradigm – to significantly improve the precision and scope of feature discovery. This exploration isn’t simply about identifying more features, but about uncovering subtle and complex relationships within data that existing methods might overlook, ultimately leading to more robust privacy protections and unlocking a broader range of applications where sensitive information must be handled with the utmost care.

The pursuit of understanding complex systems, as demonstrated by Embedding-Aware Feature Discovery, inherently involves a degree of controlled deconstruction. This framework doesn’t simply accept learned embeddings as opaque representations; it actively seeks to bridge them with interpretable, structured features. This mirrors a hacker’s mindset – taking something seemingly complete and dissecting it to reveal its underlying mechanisms. As Brian Kernighan aptly stated, “Debugging is like being the detective in a crime movie where you are also the murderer.” The EAFD framework embodies this sentiment; by probing the latent space and demanding explainability, it uncovers how event sequences function, effectively diagnosing and resolving the ‘mystery’ of their behavior, and constructing a more robust understanding from the ‘crime scene’ of data.

What Breaks Next?

The pursuit of embedding-aware feature discovery, as demonstrated by this work, inevitably exposes the brittle core of ‘interpretability’ itself. To suggest a feature is ‘discovered’ implies a prior absence, a latent potential waiting for extraction. Yet, the system defines the potential. The true challenge isn’t aligning embeddings with existing structures, but acknowledging that the structures themselves are artifacts of the encoding process. Every exploit starts with a question, not with intent; future work should focus less on finding features and more on systematically perturbing the embedding space to reveal the limits of the learned representation.

Current methodologies largely treat event sequences as static inputs. A more adversarial approach-introducing carefully crafted noise or ambiguity into the sequences-could expose vulnerabilities in the feature discovery process. Can the system differentiate genuine signals from deliberate misdirection? The framework’s reliance on LLM-based generation, while promising, introduces a dependence on the biases and limitations inherent in those models. Investigating the robustness of EAFD against adversarial LLM outputs is crucial.

Ultimately, the field needs to confront the unsettling possibility that ‘good’ features aren’t objectively true, but merely convenient approximations. The goal shouldn’t be to mirror some underlying reality, but to build systems that gracefully degrade under unexpected conditions. The next iteration of this research shouldn’t seek to refine the map, but to dismantle the territory.

Original article: https://arxiv.org/pdf/2603.15713.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/