Aligning What LLMs Do and Say: Towards Self-Consistent Explanations

TL;DR

This paper introduces PSCB, a large-scale benchmark to measure and improve the alignment between LLMs' answers and their explanations, enhancing interpretability and faithfulness.

Contribution

It presents PSCB for large-scale evaluation, finds Spearman correlation more reliable than cosine, and applies DPO to improve alignment without sacrificing accuracy.

Findings

Spearman rank correlation better indicates alignment than cosine similarity.

DPO improves alignment between answers and explanations.

Fine-tuning on attribution data outperforms supervised fine-tuning.

Abstract

Large language models (LLMs) seem to offer an easy path to interpretability: just ask them to explain their answers. Yet the features driving an answer often differ from those emphasized in its explanation, meaning post-hoc rationales can misrepresent what actually shaped the model's output. We quantify this gap by comparing the feature-importance distributions of answers and their explanations. Prior analyses reveal such discrepancies, but large-scale study has been limited by the high computational cost of attribution methods. To address this, we introduce the Post-hoc Self-Consistency Bank (PSCB), a large-scale benchmark linking model decisions with diverse explanations and attribution vectors across datasets, methods, and model families. Using PSCB, we find that Spearman rank correlation provides a more reliable signal of alignment than cosine similarity. Building on this insight, we apply Direct Preference Optimization (DPO) to attribution-based preference data, improving alignment without degrading task accuracy, and show that standard supervised fine-tuning on the same data fails to achieve comparable gains. These improvements generalize robustly across domains, paving the way toward scalable and faithful alignment between LLM decisions and their natural language explanations.