Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking

TL;DR

The Inner Thinking Transformer introduces dynamic depth scaling and adaptive computation to improve reasoning in language models without increasing parameters, achieving high performance with fewer resources.

Contribution

It proposes a novel architecture that dynamically allocates computation for critical tokens, enabling deeper reasoning without parameter growth.

Findings

Achieves 96.5% performance of larger models with fewer parameters.

Reduces training data requirements by 43.2%.

Outperforms existing Transformer variants on multiple benchmarks.

Abstract

Large language models (LLMs) face inherent performance bottlenecks under parameter constraints, particularly in processing critical tokens that demand complex reasoning. Empirical analysis reveals challenging tokens induce abrupt gradient spikes across layers, exposing architectural stress points in standard Transformers. Building on this insight, we propose Inner Thinking Transformer (ITT), which reimagines layer computations as implicit thinking steps. ITT dynamically allocates computation through Adaptive Token Routing, iteratively refines representations via Residual Thinking Connections, and distinguishes reasoning phases using Thinking Step Encoding. ITT enables deeper processing of critical tokens without parameter expansion. Evaluations across 162M-466M parameter models show ITT achieves 96.5\% performance of a 466M Transformer using only 162M parameters, reduces training data by 43.2\%, and outperforms Transformer/Loop variants in 11 benchmarks. By enabling elastic computation allocation during inference, ITT balances performance and efficiency through architecture-aware optimization of implicit thinking pathways.