Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks

TL;DR

This paper explores contrastive, LRP-based attribution as a practical interpretability tool for analyzing failures of large language models on realistic benchmarks, highlighting its utility and limitations.

Contribution

It introduces a contrastive attribution framework for LLM failure analysis, including an efficient extension for long-context inputs and a systematic empirical study.

Findings

Contrastive attribution provides informative signals in some failure cases.

The method's applicability varies across different datasets and model configurations.

Code for the framework is publicly available at https://aka.ms/Debug-XAI.

Abstract

Interpretability tools are increasingly used to analyze failures of Large Language Models (LLMs), yet prior work largely focuses on short prompts or toy settings, leaving their behavior on commonly used benchmarks underexplored. To address this gap, we study contrastive, LRP-based attribution as a practical tool for analyzing LLM failures in realistic settings. We formulate failure analysis as \textit{contrastive attribution}, attributing the logit difference between an incorrect output token and a correct alternative to input tokens and internal model states, and introduce an efficient extension that enables construction of cross-layer attribution graphs for long-context inputs. Using this framework, we conduct a systematic empirical study across benchmarks, comparing attribution patterns across datasets, model sizes, and training checkpoints. Our results show that this token-level contrastive attribution can yield informative signals in some failure cases, but is not universally applicable, highlighting both its utility and its limitations for realistic LLM failure analysis. Our code is available at: https://aka.ms/Debug-XAI.