Geometric SSM: LTI State Space Models for Selective Tasks

TL;DR

This paper demonstrates that LTI state space models, designed using geometric control principles, can achieve selectivity in sequence tasks without requiring time-varying dynamics, challenging previous assumptions.

Contribution

The authors introduce the Geometric SSM, a novel LTI model that uses geometric control to achieve selectivity with a dynamic residual generator, enabling recognition of complex patterns.

Findings

Achieves near-perfect performance on a novel extended induction head task

Maintains efficient FFT-based training

Outperforms Mamba in selectivity tasks

Abstract

A key claim in recent work on Selective State Space Models is that selectivity, the ability to focus on relevant information while filtering irrelevant inputs, requires breaking the Linear Time-Invariant (LTI) property through time-varying dynamics. We challenge this claim by demonstrating that LTI systems can achieve selectivity when designed using principles from geometric control. We introduce the Geometric SSM, in which different input patterns excite distinct invariant subspaces of the dynamics. Unlike Mamba's memoryless selection mechanism, our approach employs a dynamic residual generator that maintains temporal memory, enabling recognition of multi-token patterns without time-varying system matrices. The Geometric SSM achieves near-perfect performance on a novel extended induction head task where Mamba fails, while preserving efficient FFT-based training. Our results demonstrate that geometric control theory can inform the design of novel selective sequence models that combine theoretical rigor with practical efficiency.