Speaking the Language of Teamwork: LLM-Guided Credit Assignment in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a novel multi-agent reinforcement learning framework where large language models generate dense, agent-specific rewards from natural language descriptions, improving convergence and policy performance in sparse reward environments.

Contribution

The work presents a new LLM-guided reward generation method that enhances credit assignment in MARL, outperforming existing value decomposition approaches.

Findings

Faster convergence in MARL tasks.

Higher policy returns compared to baselines.

Effective dense reward generation from natural language.

Abstract

Credit assignment, the process of attributing credit or blame to individual agents for their contributions to a team's success or failure, remains a fundamental challenge in multi-agent reinforcement learning (MARL), particularly in environments with sparse rewards. Commonly-used approaches such as value decomposition often lead to suboptimal policies in these settings, and designing dense reward functions that align with human intuition can be complex and labor-intensive. In this work, we propose a novel framework where a large language model (LLM) generates dense, agent-specific rewards based on a natural language description of the task and the overall team goal. By learning a potential-based reward function over multiple queries, our method reduces the impact of ranking errors while allowing the LLM to evaluate each agent's contribution to the overall task. Through extensive experiments, we demonstrate that our approach achieves faster convergence and higher policy returns compared to state-of-the-art MARL baselines.