Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

TL;DR

This paper analyzes Best-of-N methods for generative model alignment, introducing a smooth variant called SBoN, and provides theoretical bounds on KL divergence and regret to improve understanding of their performance.

Contribution

It develops a theoretical framework for SBoN, deriving bounds on KL divergence and regret, and demonstrates how smoothing mitigates reward overoptimization.

Findings

SBoN reduces reward overoptimization compared to BoN.

Theoretical bounds relate sample size to divergence and regret.

Empirical results confirm smoothing benefits in low-quality proxy reward scenarios.

Abstract

A simple yet effective method for inference-time alignment of generative models is Best-of-$N$ (BoN), where $N$ outcomes are sampled from a reference policy, evaluated using a proxy reward model, and the highest-scoring one is selected. While prior work argues that BoN is almost optimal in reward vs KL tradeoffs, the effectiveness of BoN depends critically on the quality of the proxy reward model used for selection. For this purpose, we study BoN through a smooth version known as Soft Best-of-N (SBoN) and develop a theoretical framework to address this gap. We analyze the scaling behaviour of BoN by providing bounds on the KL divergence between the SBoN policy and the reference policy, offering insights into how performance varies with the number of samples. We also study the regret gap, i.e., the gap between the expected true reward under the optimal policy and the SBoN policy. Our theoretical and empirical findings show that smoothing helps SBoN mitigate reward overoptimization, especially when the quality of the proxy reward is low.