Mix&Match - Agent Curricula for Reinforcement Learning

TL;DR

Mix&Match (M&M) is a novel RL training framework that automatically creates curricula by progressively altering internal policy representations, enabling faster learning and better performance across diverse tasks and architectures.

Contribution

We introduce Mix&Match, a new curriculum learning method that modifies internal policy representations to facilitate efficient training of complex RL agents.

Findings

Achieved faster training and higher performance in a 3D first-person task.

Successfully progressed through internal architecture variants.

Improved multitask learning performance with our method.

Abstract

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.