TMAS: Scaling Test-Time Compute via Multi-Agent Synergy

TL;DR

TMAS introduces a multi-agent framework for test-time scaling of large language models, enabling collaborative reasoning and improved performance through hierarchical memories and reinforcement learning.

Contribution

It proposes a novel multi-agent synergy approach with hierarchical memories and hybrid reward training to enhance test-time compute scaling.

Findings

TMAS outperforms existing test-time scaling methods on reasoning benchmarks.

Hierarchical memories improve information reuse and reasoning efficiency.

Hybrid reward reinforcement learning stabilizes and enhances scaling performance.

Abstract

Test-time scaling has become an effective paradigm for improving the reasoning ability of large language models by allocating additional computation during inference. Recent structured approaches have further advanced this paradigm by organizing inference across multiple trajectories, refinement rounds, and verification-based feedback. However, existing structured test-time scaling methods either weakly coordinate parallel reasoning trajectories or rely on noisy historical information without explicitly deciding what should be retained and reused, limiting their ability to balance exploration and exploitation. In this work, we propose TMAS, a framework for scaling test-time compute via multi-agent synergy. TMAS organizes inference as a collaborative process among specialized agents, enabling structured information flow across agents, trajectories, and refinement iterations. To support effective cross-trajectory collaboration, TMAS introduces hierarchical memories: the experience bank reuses low-level reliable intermediate conclusions and local feedback, while the guideline bank records previously explored high-level strategies to steer subsequent rollouts away from redundant reasoning patterns. Furthermore, we design a hybrid reward reinforcement learning scheme tailored to TMAS, which jointly preserves basic reasoning capability, enhances experience utilization, and encourages exploration beyond previously attempted solution strategies. Extensive experiments on challenging reasoning benchmarks show that TMAS achieves stronger iterative scaling than existing test-time scaling baselines, with hybrid reward training further improving scaling effectiveness and stability across iterations. Code and data are available at https://github.com/IQuestLab/tmas.