MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection

TL;DR

MAVIS is a lightweight, inference-time framework that enables dynamic multi-objective alignment of large language models by combining small value models, avoiding costly fine-tuning and allowing flexible trade-offs.

Contribution

MAVIS introduces a novel inference-time alignment method using small value models for multiple objectives, improving flexibility and efficiency over traditional fine-tuning approaches.

Findings

MAVIS outperforms baselines on Pareto frontiers for multi-objective alignment.

The method enables dynamic trade-offs without modifying base model weights.

Empirical results show monotonic improvement in policy with the iterative training algorithm.

Abstract

Large Language Models (LLMs) are increasingly deployed across diverse applications that demand balancing multiple, often conflicting, objectives -- such as helpfulness, harmlessness, or humor. Many traditional methods for aligning outputs to user-specific preferences require fine-tuning models for each objective or for specific preference configurations, which is computationally expensive and inflexible. We introduce \textbf{MAVIS} -- \textit{Multi-Objective Alignment via Inference-Time Value-Guided Selection} -- a lightweight inference-time alignment framework that enables dynamic control over LLM behavior without modifying the base model's weights. MAVIS trains a set of small value models, each corresponding to a distinct objective. At inference time, these value models are combined using user-specified weights to produce a tilting function that adjusts the base model's output distribution toward desired trade-offs. The value models are trained using a simple iterative algorithm that enables monotonic improvement of the KL-regularized policy. We show empirically that MAVIS achieves a superior pareto front compared to baselines which fine-tune per-objective models and combine them post hoc or train a single preference-conditioned value model for guidance. Our code is available at https://github.com/5-Jeremy/MAVIS/tree/main.