Slot State Space Models

TL;DR

SlotSSMs introduce a modular state space framework with multiple independent slots, enhancing modeling of complex, object-centric, and long-range dependencies in sequence tasks, outperforming traditional SSMs.

Contribution

The paper proposes SlotSSMs, a new modular SSM framework that maintains separate slots with independent transitions and sparse interactions, improving sequence modeling capabilities.

Findings

Significant performance improvements in object-centric learning.

Enhanced long-range dependency modeling in visual reasoning.

Effective handling of multiple objects in video understanding.

Abstract

Recent State Space Models (SSMs) such as S4, S5, and Mamba have shown remarkable computational benefits in long-range temporal dependency modeling. However, in many sequence modeling problems, the underlying process is inherently modular and it is of interest to have inductive biases that mimic this modular structure. In this paper, we introduce SlotSSMs, a novel framework for incorporating independent mechanisms into SSMs to preserve or encourage separation of information. Unlike conventional SSMs that maintain a monolithic state vector, SlotSSMs maintains the state as a collection of multiple vectors called slots. Crucially, the state transitions are performed independently per slot with sparse interactions across slots implemented via the bottleneck of self-attention. In experiments, we evaluate our model in object-centric learning, 3D visual reasoning, and long-context video understanding tasks, which involve modeling multiple objects and their long-range temporal dependencies. We find that our proposed design offers substantial performance gains over existing sequence modeling methods. Project page is available at https://slotssms.github.io/