Memba: Membrane-driven Parameter-Efficient Fine-Tuning for Mamba

TL;DR

Memba is a novel parameter-efficient fine-tuning method for Mamba, a state space model, that uses bio-inspired LIM neurons to enhance temporal processing and improve performance on language and vision tasks.

Contribution

Memba introduces LIM neurons and cross-layer membrane transfer to adapt PEFT specifically for Mamba, addressing its unique temporal dynamics.

Findings

Significant performance improvements over existing PEFT methods.

Effective across both language and vision tasks.

Code availability facilitates reproducibility.

Abstract

State Space Models (SSMs) have emerged as powerful alternatives to attention-based Transformers, with Mamba demonstrating impressive efficiency and scalability. As these models grow increasingly larger, the need for Parameter-Efficient Fine-Tuning (PEFT) methods becomes critical to adapt pre-trained Mamba to downstream tasks without prohibitive computational costs. However, previous approaches simply apply traditional Transformer-tailored PEFT methods without addressing the unique temporal processing dynamics of SSMs. To address this limitation, we propose Memba, a membrane-driven PEFT approach specifically designed for Mamba. Memba introduces Leaky Integrate Membrane (LIM) neurons as bio-inspired gating mechanisms that naturally accumulate membrane potentials over time, enhancing selective information retention. By strategically combining LIM neurons with Low-Rank Adaptations (LoRA) and cross-layer membrane transfer, our approach significantly improves Mamba's temporal modeling capabilities. Extensive experiments across language and vision tasks demonstrate that Memba achieves substantial improvements over existing PEFT methods. The code is available at https://github.com/Intelligent-Computing-Lab-Yale/Memba.