Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection

TL;DR

Routing Mamba introduces a sparse mixture-of-experts approach to scale State Space Models efficiently, achieving comparable performance to larger dense models with fewer active parameters and reduced computational costs.

Contribution

It presents Routing Mamba, a novel sparse MoE method that enhances SSM scaling by sharing routing decisions, enabling efficient long sequence modeling with fewer parameters.

Findings

Achieves similar performance to larger dense models with fewer parameters.

Demonstrates 23% FLOPS savings compared to dense Mamba.

Maintains consistent perplexity across different context lengths.

Abstract

Linear State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3x more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance.