Integrating attention into explanation frameworks for language and vision transformers

TL;DR

This paper explores how attention weights in transformers can be integrated into explanation methods for language and vision models, enhancing interpretability by providing more meaningful local and global explanations.

Contribution

It introduces two novel explanation techniques that incorporate attention weights into existing XAI frameworks for improved interpretability of transformers.

Findings

Attention weights can be effectively integrated into explanation methods.

The proposed methods outperform some existing explainability techniques.

Attention-based explanations provide meaningful insights into model behavior.

Abstract

The attention mechanism lies at the core of the transformer architecture, providing an interpretable model-internal signal that has motivated a growing interest in attention-based model explanations. Although attention weights do not directly determine model outputs, they reflect patterns of token influence that can inform and complement established explainability techniques. This work studies the potential of utilising the information encoded in attention weights to provide meaningful model explanations by integrating them into explainable AI (XAI) frameworks that target fundamentally different aspects of model behaviour. To this end, we develop two novel explanation methods applicable to both natural language processing and computer vision tasks. The first integrates attention weights into the Shapley value decomposition by redefining the characteristic function in terms of pairwise token interactions via attention weights, thus adapting this widely used game-theoretic solution concept to provide attention-driven attributions for local explanations. The second incorporates attention weights into token-level directional derivatives defined through concept activation vectors to measure concept sensitivity for global explanations. Our empirical evaluations on standard benchmarks and in a comparison study with widely used explanation methods show that attention weights can be meaningfully incorporated into the studied XAI frameworks, highlighting their value in enriching transformer explainability.