Theory of gating in recurrent neural networks

TL;DR

This paper explores how gating mechanisms in recurrent neural networks influence their dynamics, enabling flexible control over timescales and chaos, with implications for machine learning and neuroscience.

Contribution

It introduces a theoretical framework showing gating's role in controlling RNN dynamics, including timescales, memory reset, and chaos, without fine-tuning or special symmetries.

Findings

Gating enables a flexible integrator state in RNNs.

Gates can induce a discontinuous transition to chaos.

Phase diagrams map parameter choices for RNN initialization.

Abstract

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.