TARo: Token-level Adaptive Routing for LLM Test-time Alignment

TL;DR

TARo introduces a token-level adaptive routing method that enhances reasoning capabilities of frozen large language models at inference time, significantly improving performance and generalization across domains without retraining.

Contribution

The paper presents a novel token-level routing approach that enables test-time alignment for reasoning in LLMs, extending its application beyond preference alignment.

Findings

Up to +22.4% reasoning performance improvement over base models

Enhanced out-of-distribution clinical reasoning and instruction following

Effective generalization from small to large model backbones

Abstract

Large language models (LLMs) exhibit strong reasoning capabilities but typically require expensive post-training to reach high performance. Recent test-time alignment methods offer a lightweight alternative, but have been explored mainly for preference alignment rather than reasoning. To bridge this gap, we propose, Token-level Adaptive Routing (TARo), which steers frozen LLMs toward structured reasoning entirely at inference time. Specifically, we first train reward models on step-wise mathematical traces to capture fine-grained logical consistency signals, then introduce a learnable token-level router that automatically controls the guidance of the reward model to the base model. Extensive experiments show that TARo significantly improves reasoning performance by up to +22.4% over base model and +8.4% over existing token-level test-time alignment methods, while also boosting out-of-distribution clinical reasoning (MedXpertQA) and instruction following (AlpacaEval). Furthermore, TARo also generalizes from small to large backbones without retraining, extending test-time alignment from preference optimization to robust, cross-domain reasoning.