Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs

TL;DR

HETA is a new attribution method for decoder-only language models that combines semantic, second-order, and information-theoretic components to produce more faithful and interpretable token attributions.

Contribution

We introduce HETA, a novel attribution framework specifically designed for autoregressive language models, improving interpretability over existing methods.

Findings

HETA outperforms existing attribution methods in faithfulness.

HETA produces attributions better aligned with human judgments.

A new benchmark dataset for evaluating attribution in generative models is presented.

Abstract

Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.