Recurrent Independent Mechanisms

TL;DR

Recurrent Independent Mechanisms (RIMs) is a novel recurrent architecture that promotes modularity and specialization among groups of recurrent cells, enhancing generalization and robustness in dynamic environments with varying underlying causes.

Contribution

The paper introduces RIMs, a new recurrent architecture with nearly independent modules that communicate sparsely and update selectively, improving generalization over prior monolithic models.

Findings

RIMs achieve better generalization on tasks with systematic factor variations.

Specialization among RIMs leads to improved robustness.

Sparse communication enhances modular learning.

Abstract

Learning modular structures which reflect the dynamics of the environment can lead to better generalization and robustness to changes which only affect a few of the underlying causes. We propose Recurrent Independent Mechanisms (RIMs), a new recurrent architecture in which multiple groups of recurrent cells operate with nearly independent transition dynamics, communicate only sparingly through the bottleneck of attention, and are only updated at time steps where they are most relevant. We show that this leads to specialization amongst the RIMs, which in turn allows for dramatically improved generalization on tasks where some factors of variation differ systematically between training and evaluation.