Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

TL;DR

This paper explores how the depth of sequence models influences their expressivity and error bounds, using Lie algebra theory to provide a mathematical framework that explains empirical observations of model performance.

Contribution

It introduces a Lie algebraic control perspective to analyze the relationship between model depth and expressivity, deriving error bounds and validating them experimentally.

Findings

Error decreases exponentially with depth.

Lie algebraic framework characterizes expressivity limits.

Theoretical predictions match empirical results.

Abstract

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.