NeuPL: Neural Population Learning

TL;DR

NeuPL introduces a neural population learning framework that efficiently trains diverse policies for strategy games, providing convergence guarantees and enabling transfer learning, which improves performance and strategy accessibility.

Contribution

NeuPL presents a novel neural population learning method that addresses key issues in policy diversity and training efficiency in strategy games, with theoretical guarantees and empirical validation.

Findings

NeuPL achieves convergence to a population of best-responses.

Transfer learning across policies is effectively enabled within NeuPL.

Expanding the neural population increases accessibility to novel strategies.

Abstract

Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues. NeuPL offers convergence guarantees to a population of best-responses under mild assumptions. By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains. Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.