On the Expressiveness of State Space Models via Temporal Logics
Eric Alsmann, Lowejatan Noori, Martin Lange
TL;DR
This paper analyzes the expressive power of state space models (SSM) in language modeling, showing how different mechanisms affect their ability to represent complex languages and comparing them to transformers.
Contribution
It provides a formal analysis of SSM expressiveness via temporal logics, distinguishing between quantized and unbounded models, and compares their capabilities to transformers.
Findings
Quantized SSMs are limited to regular languages.
Unbounded SSMs can capture non-regular languages and counting properties.
SSM expressiveness varies significantly with gating mechanisms.
Abstract
We investigate the expressive power of state space models (SSM), which have recently emerged as a potential alternative to transformer architectures in large language models. Building on recent work, we analyse SSM expressiveness through fragments and extensions of linear temporal logic over finite traces. Our results show that the expressive capabilities of SSM vary substantially depending on the underlying gating mechanism. We further distinguish between SSM operating over fixed-width arithmetic (quantised models), whose expressive power remains within regular languages, and SSM with unbounded precision, which can capture counting properties and non-regular languages. In addition, we provide a systematic comparison between these different SSM variants and known results on transformers, thereby clarifying how the two architectures relate in terms of expressive power.
Peer Reviews
Decision·ICLR 2026 Poster
- The proofs builds on top of the temporal logic theory and circuit complexity (AC⁰/TC⁰) using established formal frameworks from transformer expressiveness research. - Bounds are tight: Diagonal SSMs’ impossibility to recognize `(aa)*` is proven via star-free language separation. - Gate constructions are explicit and computable. - Language recognition capabilities are formally verified, not empirically approximated. - Limitations (e.g., TC⁹ upper bound not yet reduced to AC⁰ for SSMs)
- Empirical validation of depth hierarchies suggested but not explored.
The work builds upon an established literature that studies the limits of expressivity of different ML architectures from a theoretical lens. The paper studies the full breadth of different SSM variants and a clear expressivity hierarchy is established. Counterexamples have also been provided where possible to show that the lower bounds are "tight" at least from the temporal logic perspective.
The primary weakness is that these results do not suggest any ways to improve ML techniques and are far removed from practice. There is a lot of room for improving the presentation. The abstract conveys the main message well but the introduction is severely lacking in details and a reader unfamiliar with formal logic/complexity theory would find it hard to comprehend. Many aspects are not provided with an explanation. For example, the introduction never discusses what it means to recognize a la
(S1) The paper is exquisite with a clean presentation. The mathematics is there, and the authors know what they are doing. (S2) The paper's expressiveness study is impactful as it sets new grounds for what SSMs can express or not. (S3) The work bridges the gap between symbolic and neural AI. For me, this kind of investigation allows us to see temporal, formal logic through a new lens. (S4) Conjecture 1 makes sense. I don't have a formal argument, but I think the provided argument is sound.
(W1) I really enjoyed the mathematical framework and arguments (with all the proofs), but I think the paper could benefit from an additional figure showcasing how such results are related (e.g., which Lemma(s) lead to which Theorem(s)). I expect this to improve the navigation of the paper's results. (W2) The layout/presentation of Figures 1, 2, and 3 should be standardized. These figures could benefit from a more descriptive caption. (W3) Some minor inconsistencies that I've identified: - li
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms · Formal Methods in Verification · Topic Modeling