Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps

TL;DR

Grad-SAM is a new gradient-based method for interpreting transformer models by analyzing self-attention maps, providing better explanations of model predictions across multiple benchmarks.

Contribution

Introduces Gradient Self-Attention Maps (Grad-SAM), a novel approach for explaining transformer predictions through gradient analysis of self-attention units.

Findings

Grad-SAM outperforms existing explanation methods on various benchmarks.

The method effectively identifies input elements that influence model predictions.

Extensive evaluations demonstrate the superiority of Grad-SAM in interpretability tasks.

Abstract

Transformer-based language models significantly advanced the state-of-the-art in many linguistic tasks. As this revolution continues, the ability to explain model predictions has become a major area of interest for the NLP community. In this work, we present Gradient Self-Attention Maps (Grad-SAM) - a novel gradient-based method that analyzes self-attention units and identifies the input elements that explain the model's prediction the best. Extensive evaluations on various benchmarks show that Grad-SAM obtains significant improvements over state-of-the-art alternatives.