Majority of the Bests: Improving Best-of-N via Bootstrapping

TL;DR

This paper introduces Majority-of-the-Bests (MoB), a new method that improves the accuracy of best-of-N output selection from large language models by estimating output distribution modes through bootstrapping, outperforming traditional methods in multiple benchmarks.

Contribution

The paper proposes MoB, a novel bootstrapping-based approach to better select the most probable output from LLMs, addressing limitations of existing best-of-N strategies.

Findings

MoB consistently outperforms BoN in 25 out of 30 experimental setups.

Experimental results across five benchmarks and multiple LLMs show significant accuracy improvements.

Theoretical analysis supports the reliability of bootstrapping for output distribution estimation.

Abstract

Sampling multiple outputs from a Large Language Model (LLM) and selecting the most frequent (Self-consistency) or highest-scoring (Best-of-N) candidate is a popular approach to achieve higher accuracy in tasks with discrete final answers. Best-of-N (BoN) selects the output with the highest reward, and with perfect rewards, it often achieves near-perfect accuracy. With imperfect rewards from reward models, however, BoN fails to reliably find the correct answer and its performance degrades drastically. We consider the distribution of BoN's outputs and highlight that, although the correct answer does not usually have a probability close to one under imperfect rewards, it is often the most likely outcome. This suggests that the mode of this distribution can be more reliably correct than a sample from it. Based on this idea, we propose Majority-of-the-Bests (MoB), a novel selection mechanism that estimates the output distribution of BoN via bootstrapping and selects its mode. Experimental results across five benchmarks, three different base LLMs, and two reward models demonstrate consistent improvements over BoN in 25 out of 30 setups. We also provide theoretical results for the consistency of the bootstrapping. MoB serves as a simple, yet strong alternative to BoN and self-consistency, and more broadly, motivates further research in more nuanced selection mechanisms.