Swimba: Switch Mamba Model Scales State Space Models

TL;DR

Swimba introduces a novel mixture-of-experts approach for state space models that enhances capacity without increasing recurrence costs, demonstrating improved performance and efficiency on benchmark tasks.

Contribution

The paper proposes Switch Mamba (Swimba), a new MoE design for SSMs that maintains computational efficiency while increasing model capacity through expert routing in parameter space.

Findings

Swimba slightly outperforms baseline models on benchmarks.

Swimba maintains similar latency and throughput under matched FLOPs.

Theoretical analysis confirms stability of MoE-parameterized SSMs.

Abstract

Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update. We study how to introduce expert specialization into selective SSMs while preserving computational efficiency. We show that MoE--SSM can refer to two designs: (1) MoE over separated SSMs, which maintains multiple state trajectories and thus scales compute with the number of experts; and (2) MoE-parameterized SSM, which mixes experts in parameter space, maintains a single state trajectory, and evaluates the recurrence once. Our method, Switch Mamba (Swimba), follows the second design by routing over expert-produced SSM streams. Theoretically, we establish well-definedness and stability for MoE-parameterized SSMs and characterize the relationship between the two designs. Empirically, we evaluate Swimba on standard benchmark tasks and measure real-time throughput and latency. Under matched FLOPs, Swimba achieves slightly better average performance than the baseline, with a small slowdown in real-time latency and throughput. Overall, these results suggest that parameter-space MoE can increase SSM capacity while keeping the dominant recurrence cost fixed.