CoMPS: Continual Meta Policy Search

TL;DR

CoMPS is a continual meta-learning method that incrementally trains on task sequences, enabling agents to learn new tasks quickly without revisiting previous tasks, outperforming prior methods in continuous control tasks.

Contribution

Introduces CoMPS, a novel continual meta-policy search algorithm that trains incrementally over task sequences without revisiting past tasks, advancing continual reinforcement learning.

Findings

CoMPS outperforms prior continual learning methods.

CoMPS surpasses off-policy meta-reinforcement algorithms.

Effective in challenging continuous control tasks.

Abstract

We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.