On the Expressive Power and Limitations of Multi-Layer SSMs

TL;DR

This paper analyzes the expressive capabilities and limitations of multi-layer state-space models, focusing on the impacts of chain-of-thought reasoning and resource tradeoffs.

Contribution

It reveals fundamental limitations of SSMs in compositional tasks and clarifies how online CoT enhances their expressiveness and resource equivalences.

Findings

Multi-layer SSMs face inherent limitations in compositional tasks.

Online CoT significantly increases the expressive power of SSMs.

Width and precision are not interchangeable resources in the base model, but become equivalent with online CoT.

Abstract

We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs.