Uncovering the Computational Roles of Nonlinearity in Sequence Modeling Using Almost-Linear RNNs

TL;DR

This paper introduces Almost Linear RNNs (AL-RNNs) to analyze the functional role of nonlinearity in sequence modeling, revealing that many operations emerge within linear regimes and that sparse nonlinearity enhances interpretability and efficiency.

Contribution

The paper presents a systematic framework using AL-RNNs to dissect when and how nonlinearity is necessary in recurrent networks, bridging theory and practical design.

Findings

Linear regimes can implement key computational primitives.

Sparse nonlinearity improves interpretability and efficiency.

Low-data and switching tasks benefit from sparse nonlinearity.

Abstract

Sequence modeling tasks across domains such as natural language processing, time series forecasting, and control require learning complex input-output mappings. Nonlinear recurrence is theoretically required for universal approximation of sequence-to-sequence functions, yet linear recurrent models often prove surprisingly effective. This raises the question of when nonlinearity is truly required. We present a framework to systematically dissect the functional role of nonlinearity in recurrent networks, identifying when it is computationally necessary and what mechanisms it enables. We address this using Almost Linear Recurrent Neural Networks (AL-RNNs), which allow recurrence nonlinearity to be gradually attenuated and decompose network dynamics into analyzable linear regimes, making computational mechanisms explicit. We illustrate the framework across diverse synthetic and real-world tasks, including classic sequence modeling benchmarks, a neuroscientific stimulus-selection task, and a multi-task suite. We demonstrate how the AL-RNN's piecewise linear structure enables identification of computational primitives such as gating, rule-based integration, and memory-dependent transients, revealing that these operations emerge within predominantly linear backbones. Across tasks, sparse nonlinearity improves interpretability by reducing and localizing nonlinear computations, promotes shared representations in multi-task settings, and reduces computational cost. Moreover, sparse nonlinearity acts as a useful inductive bias: in low-data regimes or when tasks require discrete switching between linear regimes, sparsely nonlinear models often match or exceed fully nonlinear architectures. Our findings provide a principled approach for identifying where nonlinearity is functionally necessary, guiding the design of recurrent architectures that balance performance, efficiency, and interpretability.