The Effect of Depth on the Expressivity of Deep Linear State-Space Models

TL;DR

This paper systematically analyzes how depth and width affect the expressiveness of deep linear state-space models, revealing conditions under which depth enhances capacity especially under parameter norm constraints.

Contribution

It provides a theoretical framework characterizing the influence of depth and width on expressiveness, including equivalence and differences under norm constraints, and bounds on minimal depth for representation.

Findings

Increasing depth and width are equivalent without parameter constraints.

Deep SSMs can represent large norm targets more efficiently under norm constraints.

Upper bounds on minimal depth needed for representation are derived.

Abstract

Deep state-space models (SSMs) have gained increasing popularity in sequence modelling. While there are numerous theoretical investigations of shallow SSMs, how the depth of the SSM affects its expressiveness remains a crucial problem. In this paper, we systematically investigate the role of depth and width in deep linear SSMs, aiming to characterize how they influence the expressive capacity of the architecture. First, we rigorously prove that in the absence of parameter constraints, increasing depth and increasing width are generally equivalent, provided that the parameter count remains within the same order of magnitude. However, under the assumption that the parameter norms are constrained, the effects of depth and width differ significantly. We show that a shallow linear SSM with large parameter norms can be represented by a deep linear SSM with smaller norms using a constructive method. In particular, this demonstrates that deep SSMs are more capable of representing targets with large norms than shallow SSMs under norm constraints. Finally, we derive upper bounds on the minimal depth required for a deep linear SSM to represent a given shallow linear SSM under constrained parameter norms. We also validate our theoretical results with numerical experiments