Continuity Laws for Sequential Models

TL;DR

This paper investigates whether models inspired by continuous-time formulations truly behave continuously in time and how this affects their performance on tasks with continuous temporal structure.

Contribution

It formalizes the concept of model continuity as convergence under temporal refinement and demonstrates its impact on model behavior and task performance.

Findings

S4 exhibits stable continuous behavior.

S6 can be sensitive to input amplitude and dynamics.

Task continuity correlates with model performance.

Abstract

Inductive biases influence the behavior and performance of sequential models. In this work, we study an underexplored inductive bias in sequential modeling: continuity in time. We ask a simple question: do models motivated by continuous-time formulations, such as state-space models, actually behave continuously in time, and does this translate into better performance on tasks with continuous temporal structure? To answer this, we formalize model continuity as convergence under temporal refinement, where a model is continuous if its predictions approach an underlying continuous trajectory as the temporal discretization is refined. We show that S4 exhibits stable continuous behavior, whereas S6 (the core of Mamba) can be more sensitive to input amplitude and selective dynamics, despite being derived from a continuous dynamical system. To study whether this distinction matters for learning, we also need a corresponding notion of task continuity. We therefore introduce a metric to quantify the continuity of datasets directly from their temporal structure. Across benchmarks, we find a clear empirical alignment between task continuity, model continuity, and model performance. Beyond an inductive bias, continuity also has practical consequences: we show that it enables a simple temporal subsampling strategy that improves both efficiency and performance.