Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination

TL;DR

This paper introduces Maximum Entropy Population-based training (MEP), a method to train RL agents that collaborate effectively with humans without human data, by promoting diversity and mitigating distributional shift.

Contribution

The paper proposes MEP, a novel training approach that enhances human-AI collaboration by maintaining diversity in agent populations and dynamically prioritizing training partners.

Findings

MEP outperforms existing methods like SP, PBT, TrajeDi, and FCP in Overcooked.

Agents trained with MEP show improved robustness with human partners.

Diversity promotion reduces distributional shift in human-AI collaboration.

Abstract

We study the problem of training a Reinforcement Learning (RL) agent that is collaborative with humans without using any human data. Although such agents can be obtained through self-play training, they can suffer significantly from distributional shift when paired with unencountered partners, such as humans. To mitigate this distributional shift, we propose Maximum Entropy Population-based training (MEP). In MEP, agents in the population are trained with our derived Population Entropy bonus to promote both pairwise diversity between agents and individual diversity of agents themselves, and a common best agent is trained by paring with agents in this diversified population via prioritized sampling. The prioritization is dynamically adjusted based on the training progress. We demonstrate the effectiveness of our method MEP, with comparison to Self-Play PPO (SP), Population-Based Training (PBT), Trajectory Diversity (TrajeDi), and Fictitious Co-Play (FCP) in the Overcooked game environment, with partners being human proxy models and real humans. A supplementary video showing experimental results is available at https://youtu.be/Xh-FKD0AAKE.