Quantifying the effects of environment and population diversity in multi-agent reinforcement learning

TL;DR

This paper investigates how environment and population diversity affect generalization in multi-agent reinforcement learning, showing that diversity can improve performance in novel situations but may sometimes reduce training level performance.

Contribution

It introduces a new environment-agnostic measure of behavioral diversity and systematically studies the impact of population size and intrinsic motivation on diversity and generalization.

Findings

Procedurally generated training levels improve performance on new levels.

Increased population diversity can enhance generalization in some cases.

Training with diverse co-players sometimes decreases performance on training levels.

Abstract

Generalization is a major challenge for multi-agent reinforcement learning. How well does an agent perform when placed in novel environments and in interactions with new co-players? In this paper, we investigate and quantify the relationship between generalization and diversity in the multi-agent domain. Across the range of multi-agent environments considered here, procedurally generating training levels significantly improves agent performance on held-out levels. However, agent performance on the specific levels used in training sometimes declines as a result. To better understand the effects of co-player variation, our experiments introduce a new environment-agnostic measure of behavioral diversity. Results demonstrate that population size and intrinsic motivation are both effective methods of generating greater population diversity. In turn, training with a diverse set of co-players strengthens agent performance in some (but not all) cases.