Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

TL;DR

This paper argues against anthropomorphizing intermediate tokens in language models, emphasizing that such metaphors mislead understanding and research practices, and calls for the community to stop this misleading analogy.

Contribution

The paper critically examines the metaphor of reasoning traces in language models, highlighting its potential to mislead and proposing a shift in perspective.

Findings

Anthropomorphizing intermediate tokens can mislead model interpretation.

Current reasoning trace metaphors may hinder effective model use.

The paper advocates for avoiding human-like descriptions of model processes.

Abstract

Intermediate token generation (ITG), where a model produces output before the solution, has become a standard method to improve the performance of language models on reasoning tasks. These intermediate tokens have been called \say{reasoning traces} or even \say{thoughts} -- implicitly anthropomorphizing the traces, and implying that these traces resemble steps a human might take when solving a challenging problem, and as such can provide an interpretable window into the operation of the model's thinking process to the end user. In this position paper, we present evidence that this anthropomorphization isn't a harmless metaphor, and instead is quite dangerous -- it confuses the nature of these models and how to use them effectively, and leads to questionable research. We call on the community to avoid such anthropomorphization of intermediate tokens.