Inference-time Alignment via Sparse Junction Steering

TL;DR

This paper introduces Sparse Inference time Alignment (SIA), a method that performs targeted, sparse interventions at critical points during language model generation, significantly improving alignment efficiency and reducing computational costs.

Contribution

The paper proposes a novel sparse junction steering approach that intervenes only at high entropy decision points, outperforming dense intervention methods in alignment and efficiency.

Findings

Intervening on 20-80% of tokens achieves better alignment-efficiency trade-offs.

On models like Qwen3, as few as 20% token interventions match or surpass heavily trained models.

Reduces computational cost by up to 6x while maintaining or improving alignment quality.

Abstract

Token-level steering has emerged as a pivotal approach for inference-time alignment, enabling fine grained control over large language models by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference time Alignment (SIA), which performs sparse junction steering by intervening only at critical decision points along the generation trajectory. Our key insight is that high entropy junctions mark pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, indicating the need to introduce alignment related reward signals at these points. Extensive experiments across different model families and alignment objectives show that steering only 20% to 80% of tokens achieves superior alignment-efficiency trade offs. For strong base models such as Qwen3, intervening on as few as 20% of tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search based methods such as Best-of-N, and reduces computational cost by up to 6x.