SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series

TL;DR

SiMBA introduces a simplified architecture combining Einstein FFT and Mamba blocks, achieving state-of-the-art performance in vision and time series tasks while addressing stability issues of previous SSMs.

Contribution

The paper proposes SiMBA, a novel architecture that integrates Einstein FFT for channel modeling with Mamba for sequence modeling, improving stability and performance over existing SSMs.

Findings

Outperforms existing SSMs on multiple benchmarks

Establishes new state-of-the-art on ImageNet and transfer learning datasets

Effective in both vision and time series applications

Abstract

Transformers have widely adopted attention networks for sequence mixing and MLPs for channel mixing, playing a pivotal role in achieving breakthroughs across domains. However, recent literature highlights issues with attention networks, including low inductive bias and quadratic complexity concerning input sequence length. State Space Models (SSMs) like S4 and others (Hippo, Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address the above issues to help handle longer sequence lengths. Mamba, while being the state-of-the-art SSM, has a stability issue when scaled to large networks for computer vision datasets. We propose SiMBA, a new architecture that introduces Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations and uses the Mamba block for sequence modeling. Extensive performance studies across image and time-series benchmarks demonstrate that SiMBA outperforms existing SSMs, bridging the performance gap with state-of-the-art transformers. Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet and transfer learning benchmarks such as Stanford Car and Flower as well as task learning benchmarks as well as seven time series benchmark datasets. The project page is available on this website ~\url{https://github.com/badripatro/Simba}.