IPPO Learns the Game, Not the Team: A Study on Generalization in Heterogeneous Agent Teams

TL;DR

This study examines how IPPO agents trained with self-play generalize to new heterogeneous teammates in multi-agent environments, revealing that simple IPPO training can achieve broad generalization without explicit diversity exposure.

Contribution

The paper introduces Rotating Policy Training (RPT) to enhance agent generalization in heterogeneous multi-agent settings and demonstrates that IPPO can generalize well to unseen teammate policies.

Findings

RPT exposes agents to diverse partner strategies during training.

IPPO agents perform comparably with and without teammate diversity.

Simple IPPO training achieves generalization to novel teammates.

Abstract

Multi-Agent Reinforcement Learning (MARL) is commonly deployed in settings where agents are trained via self-play with homogeneous teammates, often using parameter sharing and a single policy architecture. This opens the question: to what extent do self-play PPO agents learn general coordination strategies grounded in the underlying game, compared to overfitting to their training partners' behaviors? This paper investigates the question using the Heterogeneous Multi-Agent Challenge (HeMAC) environment, which features distinct Observer and Drone agents with complementary capabilities. We introduce Rotating Policy Training (RPT), an approach that rotates heterogeneous teammate policies of different learning algorithms during training, to expose the agent to a broader range of partner strategies. When playing alongside a withheld teammate policy (DDQN), we find that RPT achieves similar performance to a standard self-play baseline, IPPO, where all agents were trained sharing a single PPO policy. This result indicates that in this heterogeneous multi-agent setting, the IPPO baseline generalizes to novel teammate algorithms despite not experiencing teammate diversity during training. This shows that a simple IPPO baseline may possess the level of generalization to novel teammates that a diverse training regimen was designed to achieve.