Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learning

TL;DR

This paper introduces MSARL, a multi-agent reinforcement learning framework that separates reasoning and tool use into specialized small agents, improving stability and accuracy in complex problem solving.

Contribution

MSARL is a novel multi-agent framework that explicitly decouples reasoning from tool use, enhancing performance and scalability in tool-integrated AI systems.

Findings

MSARL outperforms single-agent baselines in mathematical problem solving.

Decoupling reasoning and tool use improves stability and accuracy.

The architecture generalizes across diverse tool-use tasks.

Abstract

Recent advances in multi-agent systems highlight the potential of specialized small agents that collaborate via division of labor. Existing tool-integrated reasoning systems, however, often follow a single-agent paradigm in which one large model interleaves long-horizon reasoning with precise tool operations, leading to cognitive-load interference and unstable coordination. We present MSARL, a Multi-Small-Agent Reinforcement Learning framework that explicitly decouples reasoning from tool use. In MSARL, a Reasoning Agent decomposes problems and plans tool invocations, while multiple Tool Agents specialize in specific external tools, each trained via a combination of imitation learning and reinforcement learning with role-specific rewards. On mathematical problem solving with code execution, MSARL significantly improves reasoning stability and final-answer accuracy over single-agent baselines. Moreover, the architecture generalizes to diverse tool-use tasks, demonstrating that cognitive-role decoupling with small agents is a scalable blueprint for multi-agent AI design.