Continual Model-Based Reinforcement Learning with Hypernetworks

TL;DR

HyperCRL introduces a continual learning approach for model-based reinforcement learning using task-conditional hypernetworks, enabling efficient lifelong robot learning without revisiting all past data.

Contribution

It proposes a novel continual learning method with fixed-capacity hypernetworks that efficiently models non-stationary dynamics in sequential tasks.

Findings

Outperforms existing continual learning methods with fixed-capacity networks.

Achieves competitive results with methods that store all past experiences.

Effective in robot locomotion and manipulation tasks like pushing and door opening.

Abstract

Effective planning in model-based reinforcement learning (MBRL) and model-predictive control (MPC) relies on the accuracy of the learned dynamics model. In many instances of MBRL and MPC, this model is assumed to be stationary and is periodically re-trained from scratch on state transition experience collected from the beginning of environment interactions. This implies that the time required to train the dynamics model - and the pause required between plan executions - grows linearly with the size of the collected experience. We argue that this is too slow for lifelong robot learning and propose HyperCRL, a method that continually learns the encountered dynamics in a sequence of tasks using task-conditional hypernetworks. Our method has three main attributes: first, it includes dynamics learning sessions that do not revisit training data from previous tasks, so it only needs to store the most recent fixed-size portion of the state transition experience; second, it uses fixed-capacity hypernetworks to represent non-stationary and task-aware dynamics; third, it outperforms existing continual learning alternatives that rely on fixed-capacity networks, and does competitively with baselines that remember an ever increasing coreset of past experience. We show that HyperCRL is effective in continual model-based reinforcement learning in robot locomotion and manipulation scenarios, such as tasks involving pushing and door opening. Our project website with videos is at this link https://rvl.cs.toronto.edu/blog/2020/hypercrl