Predictability Enables Parallelization of Nonlinear State Space Models

TL;DR

This paper links the predictability of nonlinear state space models, measured by Lyapunov exponents, to their parallelizability, showing predictable systems can be evaluated in significantly fewer steps using optimization-based methods.

Contribution

It establishes a theoretical relationship between system dynamics and optimization problem conditioning, guiding the design of models suitable for parallel evaluation.

Findings

Predictable systems have well-conditioned optimization problems.

Chaotic systems exhibit poor conditioning, hindering parallelization.

Evaluation time for predictable systems scales as O((log T)^2).

Abstract

The rise of parallel computing hardware has made it increasingly important to understand which nonlinear state space models can be efficiently parallelized. Recent advances like DEER (arXiv:2309.12252) and DeepPCR (arXiv:2309.16318) recast sequential evaluation as a parallelizable optimization problem, sometimes yielding dramatic speedups. However, the factors governing the difficulty of these optimization problems remained unclear, limiting broader adoption. In this work, we establish a precise relationship between a system's dynamics and the conditioning of its corresponding optimization problem, as measured by its Polyak-Lojasiewicz (PL) constant. We show that the predictability of a system, defined as the degree to which small perturbations in state influence future behavior and quantified by the largest Lyapunov exponent (LLE), impacts the number of optimization steps required for evaluation. For predictable systems, the state trajectory can be computed in at worst $O((\log T)^2)$ time, where $T$ is the sequence length: a major improvement over the conventional sequential approach. In contrast, chaotic or unpredictable systems exhibit poor conditioning, with the consequence that parallel evaluation converges too slowly to be useful. Importantly, our theoretical analysis shows that predictable systems always yield well-conditioned optimization problems, whereas unpredictable systems lead to severe conditioning degradation. We validate our claims through extensive experiments, providing practical guidance on when nonlinear dynamical systems can be efficiently parallelized. We highlight predictability as a key design principle for parallelizable models.