Refining Answer Distributions for Improved Large Language Model Reasoning

TL;DR

This paper introduces Refined Answer Distributions, a new iterative sampling method that improves large language model reasoning by better combining multiple responses to identify the most probable answer.

Contribution

It proposes a novel algorithmic framework that enhances LLM reasoning by iteratively approximating answer distributions to find the most likely solution.

Findings

Outperforms existing combination strategies like self-consistency.

Effectively identifies the mode of answer distributions.

Demonstrates superior performance on reasoning benchmarks.

Abstract

Large Language Models (LLMs) have exhibited an impressive capability to perform reasoning tasks, especially if they are encouraged to generate a sequence of intermediate steps. Reasoning performance can be improved by suitably combining multiple LLM responses, generated either in parallel in a single query, or via sequential interactions with LLMs throughout the reasoning process. Existing strategies for combination, such as self-consistency and progressive-hint-prompting, make inefficient usage of the LLM responses. We present Refined Answer Distributions, a novel and principled algorithmic framework to enhance the reasoning capabilities of LLMs. Our approach can be viewed as an iterative sampling strategy for forming a Monte Carlo approximation of an underlying distribution of answers, with the goal of identifying the mode -- the most likely answer. Empirical evaluation on several reasoning benchmarks demonstrates the superiority of the proposed approach.