Undermining AI’s Explanations: A New Attack on Interpretable Models

by

Author: Denis Avetisyan

Researchers have demonstrated a subtle adversarial attack that manipulates how AI explains its decisions, raising concerns about the reliability of explainable AI techniques.

A novel adversarial attack subtly manipulates image classifications by injecting perturbations—derived from a competing class and masked to preserve visual fidelity—into the original image via a weighted sum, resulting in a corrupted image that remains imperceptible to humans yet yields altered explanatory attributions within the classifier, demonstrating a vulnerability in post-hoc explainability methods and highlighting the potential to mislead interpretability techniques.
A novel adversarial attack subtly manipulates image classifications by injecting perturbations—derived from a competing class and masked to preserve visual fidelity—into the original image via a weighted sum, resulting in a corrupted image that remains imperceptible to humans yet yields altered explanatory attributions within the classifier, demonstrating a vulnerability in post-hoc explainability methods and highlighting the potential to mislead interpretability techniques.

This work introduces eXIAA, a black-box attack that alters images to disrupt feature attribution maps without significantly impacting prediction accuracy, revealing vulnerabilities in post-hoc explainability methods.

Despite growing reliance on explainable AI (XAI) for model transparency, current methods remain vulnerable to subtle manipulation. This paper introduces eXIAA: eXplainable Injections for Adversarial Attack, a novel black-box attack that modifies images to drastically alter XAI-generated explanations—such as saliency maps—without impacting predictive accuracy. Our approach demonstrates that these explanations can be easily compromised, revealing a critical weakness in current interpretability techniques. This raises a fundamental question: can we truly trust explanations if they are susceptible to undetectable adversarial manipulation, particularly in high-stakes applications?

The Illusion of Transparency: Exposing the Vulnerability of Explainable AI

Deep learning models have achieved remarkable success in diverse applications, yet their internal workings often remain opaque – a characteristic commonly referred to as the ‘black box’ problem. This lack of transparency stems from the complex, multi-layered architecture of these models, making it difficult to discern why a particular input leads to a specific output. While predictive accuracy is often prioritized, this opacity hinders trust, particularly in critical domains like healthcare or finance, where understanding the rationale behind a decision is paramount. Furthermore, the inability to interpret model behavior complicates accountability, making it challenging to identify and rectify biases or errors embedded within the system. Consequently, the ‘black box’ nature of deep learning presents a significant barrier to widespread adoption and responsible implementation, prompting research into methods for increasing model interpretability.

As deep learning models gained prominence, their inherent opacity – often described as ‘black boxes’ – prompted the development of Explainable AI (XAI). XAI seeks to illuminate the reasoning behind a model’s decisions, fostering trust and enabling accountability. A core technique within XAI is Feature Attribution, which aims to quantify the influence of each input feature on the model’s output. By assigning importance scores to individual features, these methods provide insights into why a model made a particular prediction. For example, in image recognition, feature attribution might highlight the specific pixels that contributed most to classifying an image as a cat. This allows developers and users to not only understand the model’s logic but also to verify its behavior and identify potential biases or errors.

The increasing dependence on Explainable AI (XAI) methods, while intended to foster trust, introduces unforeseen vulnerabilities to machine learning systems. Recent research demonstrates that subtle manipulations can significantly alter the explanations generated by these techniques, without necessarily affecting the model’s core predictions. This ability to induce substantial changes in feature attribution – as evidenced by the data presented in Figures 2 through 5, and detailed in Appendix A.1 – opens avenues for malicious actors to mislead or deceive those relying on XAI for critical decision-making. Such manipulations could, for example, mask the true factors driving a model’s output, leading to flawed interpretations and potentially harmful consequences, highlighting a crucial need for robust validation and defense mechanisms within XAI frameworks.

Varying the parameter α significantly alters explanation stability, as demonstrated by changes in explanation scores for different classifier/explainability method pairs across varying top-k values, with dotted lines indicating baseline performance.
Varying the parameter α significantly alters explanation stability, as demonstrated by changes in explanation scores for different classifier/explainability method pairs across varying top-k values, with dotted lines indicating baseline performance.

Beyond Prediction: The Emergence of Explanation-Targeted Adversarial Attacks

Historically, adversarial attacks focused on causing machine learning models to make incorrect predictions. Current research demonstrates a shift towards attacks that manipulate the explanations generated by Explainable AI (XAI) methods, even while maintaining high prediction accuracy. This means an attacker can subtly alter the features a model highlights as important for its decision, creating a misleading rationale without necessarily changing the outcome. These attacks do not aim to make the model fail, but rather to distort the user’s understanding of why the model made a specific decision, potentially leading to inappropriate trust or flawed interpretations of model behavior. This represents a new vulnerability as users increasingly rely on XAI to validate and interpret model outputs.

Adversarial attacks are now being designed to manipulate the explanations generated by Explainable AI (XAI) methods, rather than directly altering model predictions. These attacks utilize minimal perturbations – small, carefully crafted changes to the input data – that demonstrably shift the XAI-generated explanation without substantially impacting the model’s predictive accuracy. Empirical results indicate that prediction confidence decreases by less than 10% in the majority of tested scenarios following these perturbations, suggesting a high degree of stealth. The intent is to subtly mislead users regarding the model’s reasoning process, even while maintaining correct predictions.

Attacks designed to manipulate model explanations, rather than predictions, represent a critical security vulnerability as they can induce users to incorrectly understand a model’s decision-making process and foster inappropriate reliance on flawed rationale. Evaluations demonstrate that this approach to adversarial manipulation consistently surpasses the performance of traditional attacks, particularly when applied to transformer-based architectures. The efficacy of this method is further enhanced through the strategic selection of images used to generate the adversarial perturbations, resulting in a higher rate of successful explanation manipulation with minimal impact on prediction confidence—typically less than a 10% reduction—thereby masking the attack and increasing the risk of user deception.

Confidence in the predicted class decreases consistently with increasing corruption, mirroring the trend observed in Figure 2.
Confidence in the predicted class decreases consistently with increasing corruption, mirroring the trend observed in Figure 2.

Deconstructing the Attack: Methodologies and Metrics for Assessing Explanation Robustness

Black-box and model-agnostic attacks represent a significant threat to the trustworthiness of Explainable AI (XAI) systems by demonstrating the capacity to manipulate explanations without requiring detailed knowledge of the target model’s architecture or parameters. These attacks operate by querying the model with perturbed inputs and observing the resulting changes in both predictions and explanations. The success of these approaches indicates that vulnerabilities exist that are independent of specific model designs, implying a broader systemic risk to XAI. Consequently, even models considered robust to traditional attacks may be susceptible to manipulation via explanation modification, highlighting the need for defense mechanisms that do not rely on complete model knowledge.

One-step attack methods represent an efficient means of manipulating explanations generated by Explainable AI (XAI) techniques with limited perturbation to the input data. These attacks achieve alteration of the explanation – typically saliency maps or feature attributions – in a single optimization step, requiring minimal computational resources. This efficiency demonstrates a vulnerability in current XAI methods, as even small, carefully crafted perturbations can significantly alter the explanation without necessarily impacting the model’s predictive performance. The ease with which explanations can be modified suggests a fragility in their reliability as indicators of model reasoning and raises concerns about their use in high-stakes decision-making scenarios where trustworthiness is paramount.

Attack success is quantified by assessing changes in both explanation and prediction outputs. Evaluations demonstrate substantial alterations to explanations – as detailed in Figures 2, 3, 4, 5, A.1 and supporting tables – while simultaneously preserving a high degree of predictive accuracy. This indicates that even minor perturbations can significantly impact the generated explanations without noticeably affecting model performance, revealing a potential disconnect between explanation fidelity and model robustness. The ability to induce substantial explanation change with minimal prediction change highlights the fragility of current Explainable AI (XAI) techniques and the potential for adversarial manipulation of explanation outputs.

Structural Similarity Index (SSIM) values were calculated to quantify the perceptual difference between original images and those subjected to adversarial attack. High SSIM values, as demonstrated in Figure 3, indicate that the perturbations introduced to generate the attacks are minimal and largely imperceptible to the human visual system. This suggests the attack operates with a high degree of stealth, avoiding obvious artifacts or distortions that would readily identify the manipulated input. The preservation of visual fidelity, despite successful alteration of the model’s explanation, is a key characteristic of this attack methodology.

The Implications of Deception: Contextualizing the Threat and Charting a Course for Robust Explainable AI

Despite achieving remarkable performance on image classification tasks, contemporary deep learning models—including widely adopted architectures like ResNet-18 and ViT-B16, typically pre-trained on expansive datasets such as ImageNet—exhibit a concerning vulnerability to explanation manipulation. Research indicates that subtle, often imperceptible, perturbations to input images can dramatically alter the feature attributions generated by explainable artificial intelligence (XAI) techniques. This means that explanations, intended to reveal why a model made a particular decision, can be misleading or outright fabricated, even when the model’s prediction remains correct. The susceptibility arises because these XAI methods often rely on gradients or other sensitivity analyses which are themselves vulnerable to adversarial influence, casting doubt on the trustworthiness of explanations and potentially hindering reliable model debugging or safety-critical applications.

The Running-Up Class strategy represents a focused approach to manipulating explanations in machine learning models. Rather than simply aiming to misclassify an image, this technique centers on subtly altering the input in a way that dramatically shifts which features the model deems important for its decision. By identifying the classes that, when perturbed, most strongly influence the feature attributions – the parts of the image the model ‘sees’ – the attack can precisely target and distort these explanations. This is achieved by iteratively adjusting the input image to maximize the change in feature attribution scores, effectively ‘running up’ the perceived importance of irrelevant features or diminishing the significance of crucial ones. The result isn’t necessarily a misclassification, but a fundamentally unreliable explanation of why the model made its prediction, creating a dangerous scenario where trust in the system is misplaced despite apparent accuracy.

The demonstrated susceptibility of current Explainable AI (XAI) techniques to manipulation highlights an urgent need for more resilient methodologies. Future research must prioritize the development of XAI methods less vulnerable to adversarial perturbations, potentially through the exploration of attribution techniques grounded in robust statistical principles or information theory. Simultaneously, incorporating adversarial training – a process where models are deliberately exposed to manipulated inputs during training – offers a promising avenue for enhancing explanation reliability. This proactive approach could fortify XAI systems against attacks, ensuring that the explanations generated genuinely reflect the model’s decision-making process and aren’t merely artifacts of cleverly crafted input distortions. Ultimately, a combined strategy of robust XAI development and adversarial training will be crucial for building trustworthy and dependable AI systems.

Evaluations reveal that the developed attack strategy consistently surpasses the performance of existing methods in manipulating explanations generated by XAI techniques. This superiority is particularly pronounced when targeting transformer-based architectures, where the attack demonstrates a heightened capacity to induce substantial alterations in feature attribution maps. The effectiveness of this approach hinges on a deliberate selection of attack images, carefully chosen to maximize the distortion of explanations. Supporting evidence, visually represented in Figures 2 through 5 and A.1, alongside detailed quantitative results presented in the accompanying tables, collectively demonstrate the significant and measurable changes induced in explanations, confirming the vulnerability of current XAI methods and highlighting the need for more robust explanation techniques.

Using images from a running-up class to inject perturbations yields more consistent explanations, as demonstrated by lower variance on CIFAR10 with a ResNet-18, although this approach doesn't scale to larger datasets like ImageNet due to computational cost.
Using images from a running-up class to inject perturbations yields more consistent explanations, as demonstrated by lower variance on CIFAR10 with a ResNet-18, although this approach doesn’t scale to larger datasets like ImageNet due to computational cost.

The presented work underscores a critical fragility within explainable AI, specifically concerning feature attribution methods. It demonstrates that adversarial perturbations, crafted not to alter predictions but to manipulate explanations, can effectively conceal true decision-making factors. This resonates with Tim Bern-Lee’s assertion: “The web is more a social creation than a technical one.” While the study focuses on the technical vulnerabilities of XAI, the implications are profoundly social; misleading explanations erode trust in AI systems, potentially impacting their responsible deployment. The research meticulously exposes how easily interpretations can be decoupled from actual model behavior, mirroring the web’s potential for both connection and misdirection.

What Lies Ahead?

The demonstration of subtly manipulating explanations without commensurately impacting prediction accuracy exposes a fundamental fragility. Current feature attribution methods, while offering a comforting illusion of interpretability, appear susceptible to adversarial crafting – not of the prediction itself, but of the reasoning presented to the user. This is not merely a matter of fooling a human; it suggests a disconnect between the model’s actual decision boundary and the post-hoc rationalizations it offers.

Future work must move beyond empirical validation – demonstrating that an attack works on a given dataset – toward formal guarantees. A truly robust explanation method should be provably insensitive to such manipulations, ideally grounded in mathematical principles that link explanation fidelity to model robustness. The pursuit of ‘explainable’ AI should prioritize provability, not merely post-hoc approximation. Every byte dedicated to explanation that isn’t backed by formal verification is a potential vector for deception.

The long-term implication is a reassessment of trust. If explanations are malleable, they offer little practical defense against malicious or erroneous model behavior. The field needs to confront the possibility that some models are fundamentally unexplainable in a way that provides genuine security or insight, and embrace the elegance of acknowledging that inherent limitation.

Original article: https://arxiv.org/pdf/2511.10088.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-11-16 01:19