Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments

TL;DR

This paper introduces a novel unstructured pruning method for Mamba state-space models, reducing parameters by up to 70% while maintaining over 95% of performance, enabling efficient deployment in resource-limited environments.

Contribution

It presents a gradient-aware, iterative, and global pruning framework specifically designed for Mamba models, improving efficiency without significant performance loss.

Findings

Achieved up to 70% parameter reduction with minimal performance loss

Demonstrated effectiveness across multiple benchmarks including WikiText-103 and Long Range Arena

Provided insights into Mamba architecture's redundancy and robustness

Abstract

State-space models (SSMs), particularly the Mamba architecture, have emerged as powerful alternatives to Transformers for sequence modeling, offering linear-time complexity and competitive performance across diverse tasks. However, their large parameter counts pose significant challenges for deployment in resource-constrained environments. We propose a novel unstructured pruning framework tailored for Mamba models that achieves up to 70\% parameter reduction while retaining over 95\% of the original performance. Our approach integrates three key innovations: (1) a gradient-aware magnitude pruning technique that combines weight magnitude and gradient information to identify less critical parameters, (2) an iterative pruning schedule that gradually increases sparsity to maintain model stability, and (3) a global pruning strategy that optimizes parameter allocation across the entire model. Through extensive experiments on WikiText-103, Long Range Arena, and ETT time-series benchmarks, we demonstrate significant efficiency gains with minimal performance degradation. Our analysis of pruning effects on Mamba's components reveals critical insights into the architecture's redundancy and robustness, enabling practical deployment in resource-constrained settings while broadening Mamba's applicability.