Beyond Majority Voting: Efficient Best-Of-N with Radial Consensus Score

TL;DR

The paper introduces Radial Consensus Score (RCS), a geometric, training-free method for selecting the best response from multiple LLM outputs, outperforming traditional voting methods across various benchmarks.

Contribution

RCS models semantic consensus using a weighted Fréchet mean of answer embeddings, supporting multiple weighting schemes and improving answer selection in black-box LLM settings.

Findings

RCS variants outperform strong baselines across seven benchmarks.

Performance improves with increased sampling budget.

RCS is effective in multi-agent debate and black-box scenarios.

Abstract

Large language models (LLMs) frequently generate multiple candidate responses for a given prompt, yet selecting the most reliable one remains challenging, especially when correctness diverges from surface-level majority agreement. Existing approaches, such as self-consistency, rely on discrete voting, while probability-based methods often fail to capture relationships among candidate answers or tend to underweight high-quality but less frequent responses, and do not fully leverage the geometric structure of answer representations. To address these limitations, we introduce Radial Consensus Score (RCS), a simple, efficient, and training-free method for best-of-N selection. RCS models semantic consensus by computing a weighted Fr\'echet mean (semantic center) of answer embeddings and ranking candidates by their radial distance to this center. Importantly, RCS provides a general framework that supports multiple weighting schemes, including uniform, frequency-based, and probability-based variants, enabling flexible integration of agreement signals and model confidence while remaining fully applicable in black-box settings. Extensive experiments across seven benchmarks covering short-form QA and long-form reasoning tasks, and five open-weight models, demonstrate that RCS variants consistently outperform strong baselines, with gains becoming more pronounced as the sampling budget increases. RCS also serves as an effective drop-in replacement for majority voting in multi-agent debate and exhibits strong robustness in black-box scenarios. Overall, these results highlight geometric consensus as a scalable and broadly applicable principle for reliable answer selection, extending beyond majority voting to more expressive and robust aggregation in LLM inference.