LLM Collaboration With Multi-Agent Reinforcement Learning

TL;DR

This paper models LLM collaboration as a multi-agent reinforcement learning problem and introduces MAGRPO, a new algorithm that improves cooperative response generation in LLMs.

Contribution

It proposes a novel MARL-based framework for fine-tuning LLMs to enhance collaboration, with a new algorithm MAGRPO tailored for this purpose.

Findings

MAGRPO improves LLM collaboration efficiency.

Fine-tuning with MAGRPO yields higher quality responses.

The approach demonstrates success in writing and coding tasks.

Abstract

A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.