Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMs

TL;DR

Maestro introduces a multi-agent framework with specialized exploration and synthesis roles, utilizing a novel reinforcement learning method to improve collaborative reasoning in large language models, leading to significant accuracy gains.

Contribution

The paper presents Maestro, a new multi-agent collaboration paradigm with role decoupling and a novel CLPO training method, advancing multi-agent LLM capabilities.

Findings

Achieves 6% average accuracy improvement over state-of-the-art methods.

Demonstrates effectiveness on mathematical reasoning and problem-solving benchmarks.

Outperforms existing approaches with up to 10% accuracy gains.

Abstract

Multi-agent systems (MAS) built on Large Language Models (LLMs) are being used to approach complex problems and can surpass single model inference. However, their success hinges on navigating a fundamental cognitive tension: the need to balance broad, divergent exploration of the solution space with a principled, convergent synthesis to the optimal solution. Existing paradigms often struggle to manage this duality, leading to premature consensus, error propagation, and a critical credit assignment problem that fails to distinguish between genuine reasoning and superficially plausible arguments. To resolve this core challenge, we propose the Multi-Agent Exploration-Synthesis framework Through Role Orchestration (Maestro), a principled paradigm for collaboration that structurally decouples these cognitive modes. Maestro uses a collective of parallel Execution Agents for diverse exploration and a specialized Central Agent for convergent, evaluative synthesis. To operationalize this critical synthesis phase, we introduce Conditional Listwise Policy Optimization (CLPO), a reinforcement learning objective that disentangles signals for strategic decisions and tactical rationales. By combining decision-focused policy gradients with a list-wise ranking loss over justifications, CLPO achieves clean credit assignment and stronger comparative supervision. Experiments on mathematical reasoning and general problem-solving benchmarks demonstrate that Maestro, coupled with CLPO, consistently outperforms existing state-of-the-art multi-agent approaches, delivering absolute accuracy gains of 6% on average and up to 10% at best.