Learning to Explain: Supervised Token Attribution from Transformer Attention Patterns

TL;DR

This paper introduces ExpNet, a neural network that learns to generate token importance scores from transformer attention patterns, improving interpretability in high-stakes AI applications.

Contribution

ExpNet automatically learns optimal attention feature combinations for token attribution, surpassing manual and black-box explanation methods.

Findings

ExpNet outperforms existing attention-based explanation methods.

It demonstrates strong generalization across multiple tasks.

ExpNet reduces reliance on fixed aggregation rules.

Abstract

Explainable AI (XAI) has become critical as transformer-based models are deployed in high-stakes applications including healthcare, legal systems, and financial services, where opacity hinders trust and accountability. Transformers self-attention mechanisms have proven valuable for model interpretability, with attention weights successfully used to understand model focus and behavior (Xu et al., 2015); (Wiegreffe and Pinter, 2019). However, existing attention-based explanation methods rely on manually defined aggregation strategies and fixed attribution rules (Abnar and Zuidema, 2020a); (Chefer et al., 2021), while model-agnostic approaches (LIME, SHAP) treat the model as a black box and incur significant computational costs through input perturbation. We introduce Explanation Network (ExpNet), a lightweight neural network that learns an explicit mapping from transformer attention patterns to token-level importance scores. Unlike prior methods, ExpNet discovers optimal attention feature combinations automatically rather than relying on predetermined rules. We evaluate ExpNet in a challenging cross-task setting and benchmark it against a broad spectrum of model-agnostic methods and attention-based techniques spanning four methodological families.