Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration

TL;DR

This paper introduces Tree Search-based Orchestrated Agents (TOA), a dynamic multi-agent sampling method that improves data synthesis efficiency and performance by optimizing model collaboration through Monte Carlo Tree Search.

Contribution

It presents a novel multi-step decision-making framework for multi-agent coordination using MCTS, significantly enhancing inference compute scaling and synthesis quality.

Findings

Outperforms single-agent sampling in multiple tasks.

Achieves state-of-the-art results on WMT and AlpacaEval.

Surpasses existing preference learning methods on benchmarks.

Abstract

Scaling laws for inference compute in multi-agent systems remain under-explored compared to single-agent scenarios. This work aims to bridge this gap by investigating the problem of data synthesis through multi-agent sampling, where synthetic responses are generated by sampling from multiple distinct language models. Effective model coordination is crucial for successful multi-agent collaboration. Unlike previous approaches that rely on fixed workflows, we treat model coordination as a multi-step decision-making process, optimizing generation structures dynamically for each input question. We introduce Tree Search-based Orchestrated Agents~(TOA), where the workflow evolves iteratively during the sequential sampling process. To achieve this, we leverage Monte Carlo Tree Search (MCTS), integrating a reward model to provide real-time feedback and accelerate exploration. Our experiments on alignment, machine translation, and mathematical reasoning demonstrate that multi-agent sampling significantly outperforms single-agent sampling as inference compute scales. TOA is the most compute-efficient approach, achieving SOTA performance on WMT and a 72.2\% LC win rate on AlpacaEval. Moreover, fine-tuning with our synthesized alignment data surpasses strong preference learning methods on challenging benchmarks such as Arena-Hard and AlpacaEval.