Inference-Time Language Model Alignment via Integrated Value Guidance

TL;DR

This paper introduces Integrated Value Guidance (IVG), a novel inference-time method that aligns large language models with human preferences using implicit and explicit value functions, avoiding costly fine-tuning.

Contribution

The paper presents IVG, a new inference-time alignment technique that leverages value functions at token and chunk levels, outperforming traditional fine-tuning methods.

Findings

IVG improves sentiment and summarization task alignment.

IVG enhances instruction-following benchmark performance.

Significant increase in win rates against GPT-4-Turbo with IVG.

Abstract

Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51\% \rightarrow 26.51\%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58\% \rightarrow 33.75\%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).