Modular Networks Prevent Catastrophic Interference in Model-Based Multi-Task Reinforcement Learning

TL;DR

This paper investigates how modular network structures can prevent catastrophic interference in model-based multi-task reinforcement learning, showing that isolated sub-networks improve performance over shared models.

Contribution

It demonstrates that modular networks with isolated sub-networks mitigate task confusion and enhance multi-task learning in model-based reinforcement learning.

Findings

Shared dynamics models cause task confusion and performance drops.

Modular networks with isolated sub-networks improve multi-task learning.

Results are validated on gridworld and VizDoom environments.

Abstract

In a multi-task reinforcement learning setting, the learner commonly benefits from training on multiple related tasks by exploiting similarities among them. At the same time, the trained agent is able to solve a wider range of different problems. While this effect is well documented for model-free multi-task methods, we demonstrate a detrimental effect when using a single learned dynamics model for multiple tasks. Thus, we address the fundamental question of whether model-based multi-task reinforcement learning benefits from shared dynamics models in a similar way model-free methods do from shared policy networks. Using a single dynamics model, we see clear evidence of task confusion and reduced performance. As a remedy, enforcing an internal structure for the learned dynamics model by training isolated sub-networks for each task notably improves performance while using the same amount of parameters. We illustrate our findings by comparing both methods on a simple gridworld and a more complex vizdoom multi-task experiment.