Fast and Slow Learning of Recurrent Independent Mechanisms

TL;DR

This paper introduces a modular learning framework with fast-adapting modules and slow-changing attention meta-parameters, improving out-of-distribution generalization and adaptation in reinforcement learning tasks.

Contribution

It proposes a novel training framework with dynamic module selection and meta-learning of modular components, enhancing systematic out-of-distribution generalization.

Findings

Meta-learning modular components accelerates adaptation.

Reversing parameter roles reduces performance.

System achieves faster adaptation in grid world navigation.

Abstract

Decomposing knowledge into interchangeable pieces promises a generalization advantage when there are changes in distribution. A learning agent interacting with its environment is likely to be faced with situations requiring novel combinations of existing pieces of knowledge. We hypothesize that such a decomposition of knowledge is particularly relevant for being able to generalize in a systematic manner to out-of-distribution changes. To study these ideas, we propose a particular training framework in which we assume that the pieces of knowledge an agent needs and its reward function are stationary and can be re-used across tasks. An attention mechanism dynamically selects which modules can be adapted to the current task, and the parameters of the selected modules are allowed to change quickly as the learner is confronted with variations in what it experiences, while the parameters of the attention mechanisms act as stable, slowly changing, meta-parameters. We focus on pieces of knowledge captured by an ensemble of modules sparsely communicating with each other via a bottleneck of attention. We find that meta-learning the modular aspects of the proposed system greatly helps in achieving faster adaptation in a reinforcement learning setup involving navigation in a partially observed grid world with image-level input. We also find that reversing the role of parameters and meta-parameters does not work nearly as well, suggesting a particular role for fast adaptation of the dynamically selected modules.