On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach

TL;DR

This paper investigates the expressivity of selective state-space layers in the Mamba architecture, demonstrating through polynomial analysis that they surpass linear transformers in representational power without losing generalization, supported by empirical validation.

Contribution

It provides a theoretical analysis showing selective state-space layers are more expressive than linear transformers, backed by empirical experiments across datasets.

Findings

Selective state-space layers surpass linear transformers in expressiveness.

Mamba achieves superior representational power for long sequences.

Empirical results validate theoretical insights.

Abstract

Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba's empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets.