Beyond Dense States: Elevating Sparse Transcoders to Active Operators for Latent Reasoning

TL;DR

LSTR introduces a framework that transforms sparse semantic features into active reasoning operators, enhancing interpretability and control in latent reasoning without sacrificing accuracy.

Contribution

It proposes a novel latent reasoning framework that elevates sparse transcoders into active operators, improving interpretability and controllability in reasoning models.

Findings

LSTR maintains reasoning accuracy comparable to dense models.

Sparse features in LSTR are causally effective in reasoning.

LSTR significantly improves interpretability over dense latent baselines.

Abstract

Latent reasoning compresses the chain-of-thought (CoT) into continuous hidden states, yet existing methods rely on dense latent transitions that remain difficult to interpret and control. Meanwhile, sparse representation models uncover human-interpretable semantic features but remain largely confined to post-hoc analysis. We reconcile this tension by proposing LSTR (Latent Sparse Transcoder Reasoning), a latent reasoning framework that elevates functional sparse transcoders into active reasoning operators to perform multi-step computation through sparse semantic transitions. At its core, LSTR employs a Latent Transition Transcoder (LTT) with a residual skip architecture that decouples linear manifold transport from sparse semantic updates, enabling controllable semantic resolution via explicit sparsity constraints. Extensive experiments show that LSTR preserves reasoning accuracy and compression efficiency while substantially improving interpretability over dense latent baselines. Causal interventions and trajectory analyses further demonstrate that these sparse features act as both interpretable and causally effective operators in the reasoning process.