LLMs Should Not Yet Be Credited with Decision Explanation

TL;DR

This paper argues that LLMs should not yet be credited with providing genuine decision explanations, emphasizing the need for stricter standards to distinguish explanation from prediction and rationalization.

Contribution

It introduces a bridge standard for decision-explanation credit, clarifies the distinction between prediction, rationalization, and explanation, and advocates for calibrated attribution of explanatory claims to LLMs.

Findings

Most evidence supports LLMs' decision prediction and rationale generation, not explanation.

A proposed bridge standard for decision explanation emphasizes targeted, discriminative, and process-sensitive validation.

Lack of strict standards risks premature attribution of explanatory power to LLMs.

Abstract

This position paper argues that LLMs should not yet be credited with decision explanation. This matters because recent work increasingly treats accurate behavioral prediction, plausible rationales, and outcome-conditioned reasoning traces as evidence that LLMs explain why people decide as they do, risking a premature redefinition of what counts as explanatory progress in human decision modeling. We first distinguish three claims with different evidential burdens: decision prediction, rationale generation, and decision explanation. We then argue that the evidence most commonly offered for LLM-based decision accounts directly supports the first two claims, and sometimes explanatory hypothesis generation, but does not distinguish decision explanation from prediction-supportive rationalization. Next, we propose a bridge standard for decision-explanation credit: stronger claims should specify explanatory targets, discriminate against weaker rationalizer alternatives, use target-appropriate process- or intervention-sensitive validation, and bound their scope. We then situate this standard against competing views and related literatures, clarifying why it preserves the value of LLMs as predictors, narrators, and hypothesis generators while resisting premature explanatory credit. We conclude with a principle of credit calibration: LLMs should be credited for the strongest claim their evidence warrants, and no stronger; if adopted, this principle can help turn LLMs from persuasive narrators of decisions into more reliable instruments for discovering, testing, and communicating explanations of human behavior.