When Is Diversity Rewarded in Cooperative Multi-Agent Learning?

TL;DR

This paper investigates the conditions under which diversity among agents in multi-agent systems leads to higher rewards, combining theoretical analysis with reinforcement learning experiments to identify when heterogeneity is beneficial.

Contribution

It provides a theoretical framework linking reward structure curvature to heterogeneity benefits and introduces HetGPS, a method to find scenarios where diversity improves outcomes in MARL.

Findings

Curvature of reward aggregation operators determines heterogeneity benefits.

Convexity of reward functions simplifies the assessment of diversity advantages.

HetGPS successfully identifies scenarios where heterogeneity enhances rewards.

Abstract

The success of teams in robotics, nature, and society often depends on the division of labor among diverse specialists; however, a principled explanation for when such diversity surpasses a homogeneous team is still missing. Focusing on multi-agent task allocation problems, we study this question from the perspective of reward design: what kinds of objectives are best suited for heterogeneous teams? We first consider an instantaneous, non-spatial setting where the global reward is built by two generalized aggregation operators: an inner operator that maps the $N$ agents' effort allocations on individual tasks to a task score, and an outer operator that merges the $M$ task scores into the global team reward. We prove that the curvature of these operators determines whether heterogeneity can increase reward, and that for broad reward families this collapses to a simple convexity test. Next, we ask what incentivizes heterogeneity to emerge when embodied, time-extended agents must learn an effort allocation policy. To study heterogeneity in such settings, we use multi-agent reinforcement learning (MARL) as our computational paradigm, and introduce Heterogeneity Gain Parameter Search (HetGPS), a gradient-based algorithm that optimizes the parameter space of underspecified MARL environments to find scenarios where heterogeneity is advantageous. Across different environments, we show that HetGPS rediscovers the reward regimes predicted by our theory to maximize the advantage of heterogeneity, both validating HetGPS and connecting our theoretical insights to reward design in MARL. Together, these results help us understand when behavioral diversity delivers a measurable benefit.