STARS: Synchronous Token Alignment for Robust Supervision in Large Language Models

TL;DR

STARS introduces a decoding-time algorithm for large language models that improves alignment reliability and efficiency by enforcing verification at fixed intervals, outperforming existing methods.

Contribution

The paper proposes STARS, a novel fixed-horizon, synchronous token alignment method that enhances robustness and scalability of LLM alignment during inference.

Findings

Achieves competitive alignment quality with state-of-the-art dynamic methods.

Strictly bounds rejection costs and maximizes throughput.

Outperforms fine-tuning and other inference-time strategies on HH-RLHF benchmark.

Abstract

Aligning large language models (LLMs) with human values is crucial for safe deployment. Inference-time techniques offer granular control over generation; however, they rely on model uncertainty, meaning an internal estimate of how likely the model believes its next tokens or outputs are correct, for segmentation. We show that this introduces two critical limitations: (a) vulnerability to miscalibrated confident hallucinations and (b) poor hardware utilization due to asynchronous, ragged batch processing. Together, these issues reduce alignment reliability while increasing token and compute costs, which limits their practical scalability. To address these limitations, building on dynamic inference-time alignment methods, we introduce STARS, Synchronous Token Alignment for Robust Supervision, a decoding-time algorithm, which steers generation by enforcing verification at fixed-horizon intervals. By decoupling segmentation from confidence, STARS enables lockstep parallel execution and robustly detects errors that uncertainty metrics miss. On the HH-RLHF benchmark, we demonstrate that STARS achieves competitive alignment quality with that of state-of-the-art dynamic methods, while strictly bounding rejection costs and maximizing system throughput. Furthermore, it outperforms fine-tuning and several state-of-the-art inference-time decoding strategies by good margins, and establishes fixed-horizon sampling as a robust, system-efficient alternative for aligning LLMs at scale. The code is publicly available at https://github.com/purseclab/STARS.