On the Notion that Language Models Reason

TL;DR

This paper critically examines the concept of reasoning in language models, arguing that their outputs are better understood as statistical regularities rather than genuine logical reasoning, which impacts how we evaluate their epistemic uncertainty.

Contribution

The paper clarifies the mismatch between reasoning definitions and how language models operate, proposing a view of LMs as implicit finite-order Markov kernels rather than true reasoners.

Findings

LM outputs reflect statistical regularities, not logical reasoning

Reasoning-like behavior arises from learned statistical invariances

Understanding LMs as pattern matchers affects epistemic uncertainty evaluation

Abstract

Language models (LMs) are said to be exhibiting reasoning, but what does this entail? We assess definitions of reasoning and how key papers in the field of natural language processing (NLP) use the notion and argue that the definitions provided are not consistent with how LMs are trained, process information, and generate new tokens. To illustrate this incommensurability we assume the view that transformer-based LMs implement an \textit{implicit} finite-order Markov kernel mapping contexts to conditional token distributions. In this view, reasoning-like outputs correspond to statistical regularities and approximate statistical invariances in the learned kernel rather than the implementation of explicit logical mechanisms. This view is illustrative of the claim that LMs are "statistical pattern matchers"" and not genuine reasoners and provides a perspective that clarifies why reasoning-like outputs arise in LMs without any guarantees of logical consistency. This distinction is fundamental to how epistemic uncertainty is evaluated in LMs. We invite a discussion on the importance of how the computational processes of the systems we build and analyze in NLP research are described.