DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

TL;DR

DARC is a novel inference-time method that improves alignment robustness by explicitly managing disagreement and risk during response selection, without retraining, using a KL-robust optimization approach.

Contribution

It introduces a retraining-free, risk-sensitive decoding method that explicitly accounts for disagreement, enhancing alignment robustness in language models.

Findings

DARC reduces disagreement and tail risk in responses.

Maintains competitive average quality under noisy feedback.

Provides explicit risk controls during inference.

Abstract

Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**, a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a *KL-robust (entropic)* satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.