Measuring Reasoning Trace Legibility: Can Those Who Understand Teach?

TL;DR

This paper evaluates the legibility of reasoning traces in language models, introducing transfer utility as a measure of how well traces guide weaker models, revealing trade-offs and the need for better training incentives.

Contribution

It introduces transfer utility as a new metric for reasoning trace quality and analyzes the trade-offs between trace legibility and model performance.

Findings

High-performing models have less legible traces.

Trade-offs exist between trace length and transfer utility.

Reward models do not inherently promote trace legibility.

Abstract

Language models are increasingly being trained to "reason" before answering users' queries, outputting hundreds or even thousands of tokens worth of deliberation before their final answer. While the main intention of reasoning is to improve models' ability to arrive at a correct answer, we argue that these models should be assessed for the legibility of their reasoning traces in addition to the correctness of their final answers. In this paper, we evaluate 90k traces from 12 Reasoning Language Models (RLMs) for the quality of their reasoning traces. We introduce the concept of transfer utility, which assesses how useful an RLM's reasoning traces are for guiding a weaker, non-reasoning model toward arriving at the correct answer. We find that the reasoning traces of the highest-performing models rank among the lowest for legibility. Furthermore, we uncover tensions between efficiency-based measurements of legibility (such as trace length) and transfer utility. These tensions establish a legibility Pareto frontier, and we demonstrate that an RLM's ability to output highly legible traces can be a task- and audience-dependent goal. Crucially, we find that reward models used to train RLMs do not intrinsically reward legibility. Together, these metrics and the findings they surface chart a path towards scaffolding reasoning traces for a multi-agent future.