On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era

TL;DR

This survey reviews recent advances in recurrent and transformer-based models for processing long sequences, highlighting hybrid approaches, new research opportunities, and the shift towards infinite-length sequence learning in online settings.

Contribution

It provides a comprehensive overview of the resurgence of recurrent models, hybrid architectures, and new research directions in long sequence learning during the transformer era.

Findings

Hybrid models combining Transformers and Recurrent Nets are emerging.

Deep Space-State Models offer robust approaches to sequential function approximation.

New research opportunities arise in lifelong online learning with potentially infinite sequences.

Abstract

A longstanding challenge for the Machine Learning community is the one of developing models that are capable of processing and learning from very long sequences of data. The outstanding results of Transformers-based networks (e.g., Large Language Models) promotes the idea of parallel attention as the key to succeed in such a challenge, obfuscating the role of classic sequential processing of Recurrent Models. However, in the last few years, researchers who were concerned by the quadratic complexity of self-attention have been proposing a novel wave of neural models, which gets the best from the two worlds, i.e., Transformers and Recurrent Nets. Meanwhile, Deep Space-State Models emerged as robust approaches to function approximation over time, thus opening a new perspective in learning from sequential data, followed by many people in the field and exploited to implement a special class of (linear) Recurrent Neural Networks. This survey is aimed at providing an overview of these trends framed under the unifying umbrella of Recurrence. Moreover, it emphasizes novel research opportunities that become prominent when abandoning the idea of processing long sequences whose length is known-in-advance for the more realistic setting of potentially infinite-length sequences, thus intersecting the field of lifelong-online learning from streamed data.