mGRADE: Minimal Recurrent Gating Meets Delay Convolutions for Lightweight Sequence Modeling

TL;DR

mGRADE is a hybrid-memory sequence model combining delay embeddings and gated recurrence, achieving efficient long-range and fast dynamics modeling under strict memory constraints.

Contribution

It introduces a novel hybrid architecture with learnable delay embeddings and gating, enabling efficient multi-timescale sequence modeling on resource-limited devices.

Findings

mGRADE reduces memory usage by up to 8x compared to SOTA models.

It maintains competitive accuracy on Long-Range Arena and speech classification tasks.

The learnable spacings function as delay embeddings for fast dynamics reconstruction.

Abstract

Multi-timescale sequence modeling relies on capturing both local fast dynamics and global slow context; yet, maintaining these capabilities under the strict memory constraints common to edge devices remains an open challenge. Current State-of-the-Art models with constant memory footprints trade off long-range selectivity and high-precision modeling of fast dynamics. To overcome this trade-off within a fixed memory budget, we propose mGRADE (minimally Gated Recurrent Architecture with Delay Embedding), a hybrid-memory system that introduces inductive biases across timescales by integrating a convolution with learnable temporal spacings with a lightweight gated recurrent component. We show theoretically that the learnable spacings are equivalent to a delay embedding, enabling parameter-efficient reconstruction of partially-observed fast dynamics, while the gated recurrent component selectively maintains long-range context with minimal memory overhead. On the challenging Long-Range Arena benchmark and 35-way Google Speech Commands raw audio classification task, mGRADE reduces the memory footprint by up to a factor of 8 compared to other State-of-the-Art models, while maintaining competitive performance.