Bayesian Preference Learning for Test-Time Steerable Reward Models

TL;DR

This paper introduces Variational In-Context Reward Modeling (ICRM), a Bayesian approach that enables test-time steerability of reward models through in-context preference demonstrations, improving adaptability and performance.

Contribution

The authors propose ICRM, a novel Bayesian reward modeling method that allows test-time adaptation to unseen preferences using in-context demonstrations, with theoretical guarantees and practical benefits.

Findings

ICRM improves RM-Bench accuracy from 60.5 to 70.8 with more demonstrations.

ICRM achieves lower calibration error than a generative judge on moral dilemmas.

ICRM effectively encodes verifiable rewards, outperforming conventional RMs in math reasoning.

Abstract

Reward models are central to aligning language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, and we analyze how KL regularization mitigates reward over-optimization.