School of hard knocks: Curriculum analysis for Pommerman with a fixed computational budget

TL;DR

This paper investigates curriculum strategies for training reinforcement learning agents in Pommerman within a strict computational budget, finding that starting with diverse opponents from the beginning yields more robust policies than gradual difficulty increase.

Contribution

It demonstrates that training against all opponent policies early on outperforms curriculum-based approaches in constrained computational settings.

Findings

Early exposure to all opponent policies improves robustness.

Curriculum strategies are less effective under strict computational budgets.

Modifying environment properties impacts agent performance.

Abstract

Pommerman is a hybrid cooperative/adversarial multi-agent environment, with challenging characteristics in terms of partial observability, limited or no communication, sparse and delayed rewards, and restrictive computational time limits. This makes it a challenging environment for reinforcement learning (RL) approaches. In this paper, we focus on developing a curriculum for learning a robust and promising policy in a constrained computational budget of 100,000 games, starting from a fixed base policy (which is itself trained to imitate a noisy expert policy). All RL algorithms starting from the base policy use vanilla proximal-policy optimization (PPO) with the same reward function, and the only difference between their training is the mix and sequence of opponent policies. One expects that beginning training with simpler opponents and then gradually increasing the opponent difficulty will facilitate faster learning, leading to more robust policies compared against a baseline where all available opponent policies are introduced from the start. We test this hypothesis and show that within constrained computational budgets, it is in fact better to "learn in the school of hard knocks", i.e., against all available opponent policies nearly from the start. We also include ablation studies where we study the effect of modifying the base environment properties of ammo and bomb blast strength on the agent performance.