Change of Thought: Adaptive Test-Time Computation

TL;DR

The paper introduces SELF-Transformer, an encoder layer that adaptively refines attention weights at test time, significantly improving accuracy on benchmarks without increasing model size.

Contribution

It proposes a novel self-iterative attention mechanism that enhances expressive power of encoder Transformers through input-adaptive refinement at test time.

Findings

Up to 20% accuracy improvements on encoder benchmarks.

Achieves these gains without increasing parameter count.

Demonstrates the effectiveness of input-adaptive attention refinement.

Abstract

Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class TC0. Running a Transformer autoregressively removes that ceiling -- first in next-token prediction and, more recently, in chain-of-thought reasoning. Both regimes rely on feedback loops that decode internal states into tokens only to re-encode them in subsequent steps. While this "thinking aloud" mirrors human reasoning, biological brains iterate without externalising intermediate states as language. To boost the expressive power of encoder Transformers without resorting to token-level autoregression, we introduce the SELF-Transformer: an encoder layer that iteratively refines its own attention weights to a fixed point. Instead of producing -- in one pass -- the alignment matrix that remixes the input sequence, the SELF-Transformer iteratively updates that matrix internally, scaling test-time computation with input difficulty. This adaptivity yields up to 20\% accuracy gains on encoder-style benchmarks without increasing parameter count, demonstrating that input-adaptive alignment at test time offers substantial benefits for only a modest extra compute budget. Self-Transformers thus recover much of the expressive power of iterative reasoning while preserving the simplicity of pure encoder architectures.