Inference Time Alignment with Reward-Guided Tree Search

TL;DR

DARWIN is a novel inference-time alignment method for LLMs that uses reward-guided tree search, outperforming existing techniques and matching preference-tuned models in alignment benchmarks.

Contribution

We introduce DARWIN, a reward-guided tree search approach for inference-time alignment, demonstrating superior performance over existing methods on standard benchmarks.

Findings

Outperforms Best-of-N and ARGS on AlpacaEval 2 and MT-Bench

Achieves performance comparable to preference-tuned models

Effectively trades inference-time compute for better alignment results

Abstract

Inference-time computation methods enhance the performance of Large Language Models (LLMs) by leveraging additional computational resources to achieve superior results. Common techniques, such as Best-of-N sampling, Majority Voting, and variants of tree-search algorithms have proven to be effective in boosting the performance of LLMs. These approaches strategically trade increased computational resources for improved model responses. In this work, we proposed DARWIN, an inference-time alignment method that leverages the guidance of a reward model to achieve alignment through a reward-guided tree search. Empirical evidences indicates that our method outperforms other inference-time alignment methods such as Best-of-N and ARGS on two widely accepted alignment benchmarks AlpacaEval 2 and MT-Bench. Furthermore, we show that our inference-time approach achieves performance comparable to preference-tuned models on both benchmarks, highlighting the effectiveness of trading inference-time compute for enhanced performance during inference. We have released our codes at https://github.com/declare-lab/darwin.