Emergent Search and Backtracking in Latent Reasoning Models

TL;DR

This paper explores how latent reasoning transformers perform structured search and backtracking in continuous space during reasoning, improving accuracy by recovering from errors without explicit verbalization.

Contribution

It reveals that latent reasoning transformers spontaneously develop a structured search process with backtracking, enhancing understanding of their internal reasoning mechanisms.

Findings

Latent reasoning models exhibit a structured search with exploration, commitment, and backtracking.

Backtracking occurs in 32% of instances and improves accuracy by 34%.

The search process adapts to distractor plausibility, reducing exploration time by 54%.

Abstract

What happens when a language model thinks without words? Standard reasoning LLMs verbalize intermediate steps as chain-of-thought; latent reasoning transformers (LRTs) instead perform deliberation entirely in continuous hidden space. We investigate an LRT, decoding the model's evolving beliefs at every step on a multiple-choice QA benchmark. We find that the model spontaneously learns a structured search process in latent space. Deliberation follows a consistent trajectory: an exploration phase where probability mass spreads across candidates, tentative commitment to a frontrunner, and either convergence or backtracking. Backtracking is prevalent (32% of instances), beneficial (34% accuracy gain over non-backtracking instances), and predominantly directed away from the semantically closest distractor toward the correct answer. The search is adaptive: replacing distractors with implausible alternatives shortens exploration by 54%. Latent reasoning models achieve in activation space what chain-of-thought achieves through words: the ability to be wrong, notice, and recover.