Thinking While Listening: Fast-Slow Recurrence for Long-Horizon Sequential Modeling

TL;DR

This paper introduces a novel recurrent modeling approach that combines fast latent updates with slow observation updates to enhance long-horizon sequential understanding and out-of-distribution generalization.

Contribution

The method extends latent recurrent models with interleaved fast-slow updates, enabling stable internal structures that improve long-term coherence and generalization.

Findings

Outperforms LSTM, state space models, and Transformers on long-horizon tasks.

Facilitates learning of stable, evolving internal representations.

Enhances out-of-distribution generalization in reinforcement learning and algorithmic tasks.

Abstract

We extend the recent latent recurrent modeling to sequential input streams. By interleaving fast, recurrent latent updates with self-organizational ability between slow observation updates, our method facilitates the learning of stable internal structures that evolve alongside the input. This mechanism allows the model to maintain coherent and clustered representations over long horizons, improving out-of-distribution generalization in reinforcement learning and algorithmic tasks compared to sequential baselines such as LSTM, state space models, and Transformer variants.