Humans Perceive Wrong Narratives from AI Reasoning Texts

TL;DR

This paper investigates whether humans can accurately interpret AI-generated reasoning texts, revealing a significant gap between human understanding and the actual computational process of AI models, thus questioning their interpretability utility.

Contribution

It introduces a method to evaluate human understanding of AI reasoning steps and demonstrates a fundamental discrepancy, highlighting the need to treat reasoning texts as artifacts for further investigation.

Findings

Humans only achieved 29% accuracy in identifying causally influential reasoning steps.

Even with high agreement questions, accuracy remained low at 42%.

The results challenge the assumption that reasoning texts are straightforward interpretability tools.

Abstract

A new generation of AI models generates step-by-step reasoning text before producing an answer. This text appears to offer a human-readable window into their computation process, and is increasingly relied upon for transparency and interpretability. However, it is unclear whether human understanding of this text matches the model's actual computational process. In this paper, we investigate a necessary condition for correspondence: the ability of humans to identify which steps in a reasoning text causally influence later steps. We evaluated humans on this ability by composing questions based on counterfactual measurements and found a significant discrepancy: participant accuracy was only 29%, barely above chance (25%), and remained low (42%) even when evaluating the majority vote on questions with high agreement. Our results reveal a fundamental gap between how humans interpret reasoning texts and how models use it, challenging its utility as a simple interpretability tool. We argue that reasoning texts should be treated as an artifact to be investigated, not taken at face value, and that understanding the non-human ways these models use language is a critical research direction.