Attention Head Entropy of LLMs Predicts Answer Correctness

TL;DR

This paper introduces Head Entropy, a method that predicts answer correctness in LLMs by analyzing attention entropy patterns, demonstrating improved accuracy and generalization across domains and models.

Contribution

The paper presents Head Entropy, a novel white-box approach that uses attention entropy to predict answer correctness and generalizes well out-of-domain.

Findings

Head Entropy matches or exceeds baselines in-distribution.

It outperforms baselines by +8.5% AUROC out-of-domain.

Attention patterns over questions alone predict correctness with +17.7% AUROC.

Abstract

Large language models (LLMs) often generate plausible yet incorrect answers, posing risks in safety-critical settings such as medicine. Human evaluation is expensive, and LLM-as-judge approaches risk introducing hidden errors. Recent white-box methods detect contextual hallucinations using model internals, focusing on the localization of the attention mass, but two questions remain open: do these approaches extend to predicting answer correctness, and do they generalize out-of-domains? We introduce Head Entropy, a method that predicts answer correctness from attention entropy patterns, specifically measuring the spread of the attention mass. Using sparse logistic regression on per-head 2-Renyi entropies, Head Entropy matches or exceeds baselines in-distribution and generalizes substantially better on out-of-domains, it outperforms the closest baseline on average by +8.5% AUROC. We further show that attention patterns over the question/context alone, before answer generation, already carry predictive signal using Head Entropy with on average +17.7% AUROC over the closest baseline. We evaluate across 5 instruction-tuned LLMs and 3 QA datasets spanning general knowledge, multi-hop reasoning, and medicine.