Efficient Reasoning with Hidden Thinking

TL;DR

Heima compresses chain-of-thought reasoning in multimodal large language models into abstract tokens, maintaining reasoning quality while significantly improving efficiency, supported by theoretical analysis and empirical validation.

Contribution

This work introduces Heima, a novel CoT compression framework that preserves reasoning capabilities with minimal information loss, validated through theoretical and experimental analysis.

Findings

Heima reduces reasoning token length while maintaining accuracy.

Theoretical analysis shows reasoning capability is preserved with mutual information.

Experiments demonstrate improved efficiency and zero-shot performance.

Abstract

Chain-of-Thought (CoT) reasoning has become a powerful framework for improving complex problem-solving capabilities in Multimodal Large Language Models (MLLMs). However, the verbose nature of textual reasoning introduces significant inefficiencies. In this work, we propose Heima (as hidden llama), an effective CoT compression framework that condenses lengthy CoTs into a small set of abstract thinking tokens, preserving essential reasoning while removing redundancy. We then conduct a theoretical analysis from an information-theoretic perspective, quantifying the information gap induced by compression, showing that reasoning capability is preserved when non-trivial mutual information is retained. To further explore and quantify this information gap, we design the adaptive interpreter that maps thinking tokens back to variable-length textual sequences, thereby reconstructing the reasoning process. Experiments across diverse reasoning benchmarks demonstrate that Heima improves reasoning efficiency, while maintaining or even achieving better zero-shot accuracy. Moreover, the interpreter reconstructs coherent reasoning progresses from compressed thinking tokens, revealing that the information gap is minimal and validating the effectiveness of the proposed framework. This work paves the way for scalable latent reasoning models and advances our understanding of efficient reasoning processes in large models. Code: https://github.com/shawnricecake/Heima