Pretraining with Token-Level Adaptive Latent Chain-of-Thought

TL;DR

This paper introduces a novel pretraining method that internalizes latent Chain-of-Thought reasoning at the token level, dynamically adjusting computation per token to improve language modeling efficiency and accuracy.

Contribution

It proposes adaptive latent CoT, a method where models generate variable-length reasoning trajectories per token, reducing computation and enhancing performance without increasing model size.

Findings

Improves language modeling perplexity across benchmarks.

Reduces training and inference computation compared to prior methods.

Enhances downstream task accuracy with fewer FLOPs.

Abstract

Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-token computation without expanding parameters, by internalizing latent Chain-of-Thought (CoT) into pretraining. We propose Pretraining with Token-Level Adaptive Latent CoT (adaptive latent CoT), where the model generates a variable-length latent CoT trajectory before emitting each token -- allocating longer trajectories to difficult tokens and shorter (or even zero) trajectories to easy ones. Importantly, this behavior emerges naturally from one-stage pretraining on general text and reduces computation in both training and inference via token-wise adaptive halting. Experiments with Llama architectures show that adaptive latent CoT consistently improves language modeling perplexity and broad downstream accuracy, even with fewer training FLOPs than prior recurrent baselines.