Tracing System Secrets: Machine Learning from Kernel Call Graphs

Author: Denis Avetisyan

A new approach leverages the Linux kernel’s ftrace function graph tracer and machine learning to identify subtle system behaviors, including the detection of encryption activity.

The system orchestrates an encryption detection process, acknowledging that each step-from initial data input to final classification-introduces potential vulnerabilities, and that the entire structure is fundamentally a prediction of eventual compromise, rather than a fortress against it.

This study demonstrates the effectiveness of applying graph neural networks to ftrace function_graph data for accurate system behavior analysis and encryption detection.

Analyzing system behavior often requires navigating vast and complex datasets, yet extracting meaningful insights remains a significant challenge. This paper, ‘Exploiting \texttt{ftrace}’s \texttt{function\_graph} Tracer Features for Machine Learning: A Case Study on Encryption Detection,’ introduces a novel approach leveraging the Linux kernel’s ftrace function graph tracer to generate informative data for machine learning applications. Experiments demonstrate that graph-based features derived from function call traces achieve up to 99.28% accuracy in detecting encryption activities and effectively identifying running programs. Could this methodology unlock new avenues for proactive security analytics and performance monitoring through a deeper understanding of system-level interactions?

The System Reveals Itself: Beyond Simple Monitoring

Contemporary systems, encompassing everything from cloud infrastructure to embedded devices, are characterized by a burgeoning complexity that generates unprecedented volumes of data. This data deluge, while potentially insightful, often overwhelms traditional analytical approaches, rendering them incapable of effectively discerning meaningful patterns from noise. The sheer scale and velocity of information, coupled with intricate interdependencies between system components, obscure the root causes of performance bottlenecks and security vulnerabilities. Consequently, organizations struggle to proactively manage their systems, often resorting to reactive troubleshooting and experiencing prolonged downtime. The challenge lies not merely in collecting data, but in transforming it into actionable intelligence – a task demanding innovative approaches to data processing, visualization, and analysis capable of navigating this landscape of systemic intricacy.

Conventional system monitoring frequently relies on aggregated metrics – CPU utilization, memory consumption, network bandwidth – which, while offering a high-level overview, often fall short when diagnosing nuanced problems. These summaries obscure the critical details necessary to pinpoint the precise origin of performance bottlenecks or security breaches. For instance, a spike in CPU usage might indicate a problem, but without tracing individual function calls or identifying the specific process consuming resources, the root cause remains elusive. Similarly, detecting a network intrusion requires more than simply observing increased traffic; detailed packet inspection and behavioral analysis are essential. Consequently, organizations are increasingly recognizing the limitations of traditional approaches and seeking more granular visibility into system behavior to proactively address issues before they escalate.

To truly grasp the intricacies of modern systems, analysis must move beyond aggregated metrics and embrace detailed, low-level tracing. Traditional performance monitoring often provides a high-level overview, indicating that a problem exists, but offering little insight into why. This necessitates capturing and examining kernel-level events – system calls, interrupts, memory allocations – to reconstruct the precise sequence of operations leading to a particular outcome. By meticulously tracking these events, researchers and system administrators can move from reactive troubleshooting to proactive identification of bottlenecks, security vulnerabilities, and inefficient code paths. This granular visibility doesn’t merely diagnose issues; it unlocks a deeper understanding of system behavior, enabling optimization, improved resilience, and the development of more robust and secure applications. The shift represents a move from observing the symptoms to understanding the underlying mechanisms that drive system performance.

Effective proactive system management hinges on the ability to capture and interpret events occurring within the kernel, the core of an operating system. These kernel-level events – system calls, interrupts, memory allocations – provide a granular view of system behavior unavailable through conventional monitoring. Robust tools are therefore essential, not merely to record these events, but to correlate them, analyze patterns, and ultimately predict potential issues before they escalate. This detailed visibility allows for the identification of performance bottlenecks, the detection of malicious activity, and the optimization of resource allocation, shifting the focus from reactive troubleshooting to preventative maintenance and fostering a more resilient and efficient system architecture.

From Signals to Vectors: Mapping the System’s Language

Ftrace is a kernel-level tracing facility in Linux that provides a comprehensive mechanism for capturing system-level events, including function calls, interrupts, and scheduler activity. While highly detailed, the raw output from Ftrace consists of timestamped event records that are not readily interpretable for automated analysis. These records typically include process IDs, CPU cores, event types, and associated arguments, resulting in large volumes of unstructured data. Processing this data directly requires significant effort in parsing, filtering, and aggregating events to extract meaningful insights, making it challenging to apply analytical techniques without substantial pre-processing.

Sys2vec and Fmeter address the challenge of analyzing raw system call traces by converting them into dense vector representations. These methods utilize techniques such as word embeddings, originally developed for natural language processing, to map sequences of system calls to numerical vectors. Specifically, system calls are treated as tokens, and algorithms learn to represent each call and sequence of calls as a point in a high-dimensional vector space. This allows machine learning algorithms, which operate on numerical data, to be applied to system behavior analysis; similar system behaviors will result in vectors that are close to each other in this space, facilitating tasks like anomaly detection and performance prediction.

Converting system events into numerical vectors, a process known as embedding, facilitates the application of machine learning (ML) techniques for system analysis. These vectors represent system behavior in a quantifiable format suitable for algorithms like clustering, classification, and regression. Anomaly detection leverages these vectors to identify deviations from established baselines, signaling potential issues or security breaches. Predictive failure analysis utilizes time-series vector data to train models capable of forecasting component or system failures based on observed trends. Furthermore, performance optimization can be achieved by employing reinforcement learning or other ML methods to adjust system parameters based on vector representations of performance metrics, leading to improved resource utilization and responsiveness.

The conversion of system behavior into vector representations facilitates automated analysis by enabling the application of machine learning algorithms to identify patterns and deviations from normal operation. This capability extends to intelligent threat detection, where anomalies in system call sequences, as represented in the vector space, can indicate malicious activity or compromised system integrity. Automated analysis reduces the need for manual log review and expert intervention, while the vector-based approach allows for scalable monitoring of complex systems and rapid response to potential security incidents. Furthermore, the quantitative nature of vector embeddings allows for the development of predictive models for system failures and performance bottlenecks.

Decoding the System’s Intent: Algorithms in Action

Supervised learning algorithms, including Support Vector Machines (SVMs) and XGBoost, are employed in system intelligence to categorize system behaviors and pinpoint malicious activities through a training process. These algorithms require labeled datasets, where system actions are pre-classified as either benign or malicious, allowing the model to learn distinguishing patterns. SVMs achieve classification by identifying optimal hyperplanes that separate different classes within the feature space, while XGBoost, a gradient boosting algorithm, builds an ensemble of decision trees to improve predictive accuracy and robustness. The effectiveness of these algorithms relies on feature engineering, where relevant system characteristics-such as CPU usage, network traffic, and memory access patterns-are extracted and used as input for the classification model. Properly trained models can then classify new, unseen system behavior, flagging potential threats or anomalies for further investigation.

Recurrent Neural Networks (RNNs), specifically Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks, are well-suited for analyzing time-series data commonly found in system monitoring. Unlike traditional neural networks that treat inputs as independent, RNNs maintain an internal state, or “memory,” allowing them to process sequential information by considering prior inputs in the sequence. This capability is critical for detecting temporal patterns; for example, a GRU or LSTM can identify an unusual sequence of system calls that deviates from established baseline behavior, potentially indicating a malicious attack or the onset of a performance bottleneck. The gating mechanisms within GRUs and LSTMs address the vanishing gradient problem common in standard RNNs, enabling them to learn long-term dependencies within the sequential data and improve the accuracy of anomaly detection.

Convolutional Neural Networks (CNNs) offer a robust method for analyzing system data by leveraging their ability to automatically learn hierarchical representations of features. Unlike traditional methods requiring manual feature engineering, CNNs utilize convolutional filters to scan system data – which can be represented as time-series data or system logs – and identify patterns without predefined characteristics. These filters detect local correlations, and subsequent pooling layers reduce dimensionality while retaining important information. Multiple convolutional and pooling layers are typically stacked to extract increasingly complex features, allowing the CNN to model non-linear relationships within the system data and effectively identify anomalies or predict future behavior. The architecture is particularly effective when dealing with high-dimensional data, as it reduces the number of parameters compared to fully connected networks, thereby mitigating the risk of overfitting and improving generalization performance.

The integration of machine learning algorithms – including Support Vector Machines, XGBoost, Recurrent Neural Networks, and Convolutional Neural Networks – with vectorized system representations has yielded substantial gains in detection accuracy. Specifically, testing has demonstrated an accuracy of 99.28% in binary classification of system tasks. Vectorization transforms complex system data into a numerical format suitable for algorithmic processing, enabling these algorithms to efficiently identify patterns and anomalies. This approach significantly outperforms traditional methods reliant on manual feature engineering and threshold-based detection, providing a more robust and scalable solution for system intelligence applications.

The Architecture of Influence: Mapping System Dependencies

The Linux kernel, a complex system at the heart of countless devices, operates through intricate relationships between its various functions. The Kernel Call Graph meticulously maps these dependencies, presenting a structural overview of how functions interact and rely on one another. This graph isn’t merely a diagram; it’s a roadmap of system behavior, revealing critical pathways that handle essential operations. By visualizing these connections, researchers and developers can pinpoint functions that, if compromised, could have cascading effects on the entire system. Furthermore, the graph highlights potential vulnerabilities: functions acting as central hubs, or those with many dependencies, become prime targets for attackers seeking to disrupt system integrity. Understanding this structural view is therefore crucial for both proactive security analysis and effective debugging of kernel-level issues.

The Linux kernel, a complex system of interacting functions, benefits from analysis using graph theory to reveal the significance of individual components. Several metrics quantify this influence: Betweenness Centrality identifies functions frequently lying on the shortest paths between others, indicating control over kernel operations; Eigenvector Centrality measures a function’s influence based on the importance of the functions it connects to – a function linked to other highly central functions gains prominence; and the Clustering Coefficient assesses how interconnected a function’s neighbors are, highlighting tightly-knit, potentially critical modules. By calculating these values for each kernel function, researchers gain insight into the kernel’s structural organization, pinpointing key control points and potential bottlenecks, and ultimately understanding how changes to one function can ripple through the entire system. These metrics provide a numerical representation of functional importance, allowing for a data-driven approach to kernel security and performance optimization.

The integration of Function_Graph Tracer with graph metrics offers a powerful methodology for pinpointing crucial functions and potential security weaknesses within the Linux kernel. Function_Graph Tracer dynamically constructs a call graph during kernel execution, detailing the relationships between functions as they are invoked. This data, when coupled with metrics like Betweenness Centrality – which highlights functions acting as bridges between different parts of the kernel – and Eigenvector Centrality – which identifies functions connected to other important functions – allows for a nuanced understanding of functional importance. By analyzing these metrics, researchers can effectively prioritize functions for security auditing and identify those that, if compromised, could lead to widespread system vulnerabilities, effectively mapping potential attack vectors within the kernel’s complex architecture.

Analysis reveals a significant relationship between a function’s importance within the Linux kernel – as determined by graph centrality metrics – and the likelihood of that function being implicated in system vulnerabilities. Researchers developed models leveraging these metrics to predict vulnerabilities, achieving a robust performance of 0.858 for macro-averaged F1-score and 0.867 for micro-averaged F1-score in multi-label classification tasks. These results suggest that functions identified as central within the kernel’s call graph are disproportionately likely to be involved in security exploits, offering a promising avenue for proactive vulnerability detection and targeted security analysis. The predictive power of these models demonstrates the potential for using graph-based analysis to enhance system security and prioritize mitigation efforts.

Towards a Self-Aware System: Adaptive Security and Future Directions

Adaptive execution tracing represents a significant advancement in system monitoring by intelligently balancing the need for detailed runtime information with the preservation of system performance. Traditional tracing methods often impose a substantial overhead, particularly during peak loads, rendering them impractical for continuous operation. This novel approach dynamically adjusts the granularity of tracing based on real-time system behavior; during normal operation, tracing is minimized to reduce overhead, while it intensifies automatically in response to anomalies or suspicious activity. This responsiveness is achieved through continuous monitoring of system metrics and the implementation of feedback loops that regulate tracing intensity. Consequently, adaptive tracing provides a comprehensive view of system execution without incurring the performance penalties associated with static, high-fidelity tracing, paving the way for more robust and efficient security monitoring.

The integration of dynamic tracing technologies with machine learning algorithms heralds a shift towards predictive system security. These intelligent systems move beyond reactive threat detection by learning from real-time behavioral data gathered through kernel-level tracing and graph analysis. This allows for the identification of anomalous patterns indicative of potential attacks before they fully manifest, enabling proactive mitigation strategies. By continuously refining their understanding of normal system operation, these models can distinguish between benign activity and malicious intent with increasing accuracy, ultimately reducing the attack surface and bolstering overall system resilience. The result is a security posture that adapts to evolving threats, offering a significant advantage over traditional, signature-based approaches.

Evaluations reveal a remarkably consistent performance from the binary classification models employed in this research. Cross-validation testing demonstrates an accuracy of $99.31 \pm 0.0009$, indicating a high degree of reliability in correctly identifying system behaviors. Complementing this, the achieved F1-score of $0.9923 \pm 0.0010$ signifies an exceptional balance between precision and recall – minimizing both false positives and false negatives. These metrics collectively suggest the models possess a robust capacity to differentiate between benign and malicious activities with minimal error, offering a strong foundation for proactive security measures and intelligent system adaptation.

The fusion of kernel-level tracing, graph analysis, and machine learning represents a significant leap towards genuinely intelligent and proactive system security. By meticulously monitoring system behavior at its core, and representing that activity as a dynamic graph, researchers have enabled machine learning models to discern patterns indicative of malicious activity with unprecedented accuracy. This integrated approach recently demonstrated an Area Under the Receiver Operating Characteristic curve (ROC AUC) of 0.9994 in binary classification tasks, effectively distinguishing between benign and adversarial actions. Such high performance suggests the potential for systems that not only react to threats, but anticipate and neutralize them before they can impact system integrity, ushering in a new paradigm of resilient computing.

The pursuit of identifying encryption through function call analysis, as detailed in the study, feels less like engineering and more like tending a garden of probabilities. Each function call is a seed, its execution a stem, and the resulting graph a complex bloom-or a withered leaf, depending on the presence of cryptographic operations. Grace Hopper famously stated, “It’s easier to ask forgiveness than it is to get permission.” This sentiment resonates deeply; the researchers didn’t attempt to build a detection system, but rather allowed patterns to emerge from the system’s natural behavior, accepting a degree of false positives as the cost of uncovering hidden activity. The very act of tracing with ftrace’s function_graph acknowledges that complete control is an illusion; one observes and interprets, hoping to predict the next bloom before the frost arrives.

What Lies Ahead?

The coupling of kernel tracing with machine learning, as demonstrated here, does not offer a destination, but rather a new kind of wilderness. Each identified pattern of encrypted activity is less a solution than a temporary truce with entropy. The system reveals itself not as something to be understood, but as a growing thing, constantly differentiating itself from prior states. One suspects the most valuable outcome of this work isn’t accurate detection, but the inevitable discovery of what remains undetectable.

The current reliance on function call graphs, while insightful, is a scaffolding built upon shifting sands. The kernel is not static; it breathes, it adapts, and the very functions used as features will, with time, be refactored, renamed, or replaced. Each refinement of the model is a prayer offered to the gods of legacy code, and will, inevitably, end in repentance. The true challenge lies not in capturing a moment, but in building models resilient to constant metamorphosis.

Future work will undoubtedly explore more sophisticated graph representations, perhaps incorporating data flow or control flow information. However, the fundamental limitation remains: the system doesn’t want to be known. It yields information grudgingly, and any attempt to fully map its interior is a fool’s errand. The art, then, isn’t in building a perfect detector, but in learning to read the shadows it casts.

Original article: https://arxiv.org/pdf/2512.04590.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/