Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications

TL;DR

This paper introduces the Cumulative Memory Recurrent Unit (CMRU), a novel RNN architecture that enhances learning stability and performance for ultra-low power applications by addressing gradient blocking issues.

Contribution

It proposes a cumulative update formulation for BMRU, creating skip-connections through time, improving convergence stability and outperforming existing RNNs on various benchmarks.

Findings

CMRU matches or outperforms LRUs and minGRUs on diverse benchmarks.

The cumulative formulation improves convergence stability and reduces initialization sensitivity.

CMRU retains quantized states and noise-resilient dynamics suitable for analog hardware.

Abstract

Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the $\alpha$CMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and $\alpha$CMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.