On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study

TL;DR

This study investigates how the structure and efficiency of reasoning traces affect the generalization of language models, revealing that models trained on longer, redundant reasoning traces can outperform those trained on optimal traces.

Contribution

The paper introduces a controlled shortest-path task to analyze the impact of reasoning trace efficiency on model generalization, highlighting the importance of coherent, incremental reasoning.

Findings

Models trained on inefficient, longer traces generalize better.

Redundancy in reasoning traces can improve model performance.

Coherent, incremental reasoning traces facilitate training and generalization.

Abstract

Recent advances in natural language processing highlight two key factors for improving reasoning in large language models (LLMs): (i) allocating more test-time compute tends to help on harder problems but often introduces redundancy in the reasoning trace, and (ii) compute is most effective when reasoning is systematic and incremental, forming structured chains of thought (CoTs) akin to human problem-solving. To study these factors in isolation, we introduce a controlled setting based on shortest-path tasks in layered graphs. We train decoder-only transformers on question-trace-answer triples using a custom tokenizer, comparing models trained on optimal bottom-up dynamic programming traces with those trained on longer, valid traces involving backtracking. Surprisingly, with the same training-token budget, models trained on inefficient traces generalize better to unseen graphs. This benefit is not due to length alone-injecting arbitrary redundancy into reasoning traces fails to help and can even hurt performance. Instead, we find that generalization correlates with the model's confidence in next-token prediction, suggesting that long, coherent, and locally incremental traces make the training signal easier to optimize.