Large Language Model Guided Incentive Aware Reward Design for Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper presents an automated reward design framework using large language models to improve coordination in cooperative multi-agent reinforcement learning, reducing manual effort.

Contribution

It introduces a novel LLM-guided reward synthesis method that generates effective auxiliary rewards for multi-agent systems, evaluated across diverse environments.

Findings

Higher task returns and delivery counts achieved

Most gains in environments with interaction bottlenecks

Stronger interdependence and better signal alignment in synthesized rewards

Abstract

Designing effective auxiliary rewards for cooperative multi-agent systems remains challenging, as misaligned incentives can induce suboptimal coordination, particularly when sparse task rewards provide insufficient grounding for coordinated behavior. This study introduces an automated reward design framework that uses large language models to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and trains policies from scratch using MAPPO under a fixed computational budget. The candidates are then evaluated based on their performance, and selection across generations relies solely on the sparse task returns. The framework is evaluated in four Overcooked-AI layouts characterized by varying levels of corridor congestion, handoff dependencies, and structural asymmetries. The proposed reward design approach consistently yields higher task returns and delivery counts, with the most pronounced gains observed in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components reveals stronger interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the proposed LLM-guided reward search framework mitigates the need for manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.