Parallelizable memory recurrent units

TL;DR

This paper introduces memory recurrent units (MRUs) that combine the persistent memory of nonlinear RNNs with the parallel processing capabilities of state-space models, enabling efficient long-term sequence modeling.

Contribution

The authors propose a new family of RNNs called MRUs that leverage multistability for persistent memory while maintaining parallelizable computations, and introduce a specific implementation, BMRU.

Findings

BMRU achieves good results on long-term dependency tasks.

Hybrid networks combining MRUs and SSMs are both parallelizable and capable of transient and persistent memory.

MRUs outperform traditional SSMs in tasks requiring long-term memory.

Abstract

With the emergence of massively parallel processing units, parallelization has become a desirable property for new sequence models. The ability to parallelize the processing of sequences with respect to the sequence length during training is one of the main factors behind the uprising of the Transformer architecture. However, Transformers lack efficiency at sequence generation, as they need to reprocess all past timesteps at every generation step. Recently, state-space models (SSMs) emerged as a more efficient alternative. These new kinds of recurrent neural networks (RNNs) keep the efficient update of the RNNs while gaining parallelization by getting rid of nonlinear dynamics (or recurrence). SSMs can reach state-of-the art performance through the efficient training of potentially very large networks, but still suffer from limited representation capabilities. In particular, SSMs cannot exhibit persistent memory, or the capacity of retaining information for an infinite duration, because of their monostability. In this paper, we introduce a new family of RNNs, the memory recurrent units (MRUs), that combine the persistent memory capabilities of nonlinear RNNs with the parallelizable computations of SSMs. These units leverage multistability as a source of persistent memory, while getting rid of transient dynamics for efficient computations. We then derive a specific implementation as proof-of-concept: the bistable memory recurrent unit (BMRU). This new RNN is compatible with the parallel scan algorithm. We show that BMRU achieves good results in tasks with long-term dependencies, and can be combined with state-space models to create hybrid networks that are parallelizable and have transient dynamics as well as persistent memory.