Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning

TL;DR

This paper presents CoRL, a reinforcement learning framework for a centralized multi-agent LLM system that optimizes task performance and inference cost across different budget levels, enabling scalable and efficient collaboration.

Contribution

It introduces a novel centralized coordination approach with reinforcement learning for multi-agent LLMs, allowing cost control and performance optimization under varying budgets.

Findings

Outperforms individual experts at high budgets.

Maintains strong performance at low budgets.

Demonstrates effectiveness across diverse benchmarks.

Abstract

Large language models (LLMs) exhibit complementary strengths across domains and come with varying inference costs, motivating the design of multi-agent LLM systems where specialized models collaborate efficiently. Existing approaches predominantly rely on decentralized frameworks, which invoke multiple LLMs for every input and thus lead to substantial and uncontrolled inference costs. In this work, we introduce a centralized multi-LLM framework, where a controller LLM selectively coordinates a pool of expert models in a cost-efficient and cost-controllable manner. We formulate this coordination problem as reinforcement learning with dual objectives: maximizing task performance while minimizing the overall inference cost. In addition, we expect the multi-agent system to have adapted behavior with different budget conditions during inference. To this end, we propose CoRL, a reinforcement learning framework that optimizes the performance cost trade-off in a controllable multi-budget setting. Experiments on four diverse benchmarks demonstrate that CoRL enables a single system to surpass the best expert LLM under high-budget settings, while maintaining strong performance in more economical low-budget modes, highlighting the effectiveness of centralized coordination for scalable and cost-efficient multi-agent LLM systems.