Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation

TL;DR

This paper introduces HADES, a GSP-inspired hierarchical adaptive filter bank framework for SSMs like Mamba2, improving efficiency and interpretability while maintaining competitive performance across language tasks.

Contribution

HADES reinterprets Mamba2 as an adaptive filter bank on a line graph, introducing hierarchical filters for global and local behaviors, bridging GSP and neural sequence modeling.

Findings

HADES achieves comparable performance to Mamba2 on multiple benchmarks.

HADES uses only 58.9% of the parameters of baseline models.

HADES provides a structured, interpretable filtering approach for SSMs.

Abstract

State-space models (SSMs) offer efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, uses selective input gating and a multi-head structure, enabling parallel computation and strong benchmark performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we propose a novel method called Hierarchical ADaptive filter bank for Efficient SSMs (HADES), a Graph Signal Processing (GSP)-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. Our hierarchical architecture introduces two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved through structured bias on the parameter {\Delta}. HADES achieves comparable performance to baseline models including Mamba2 across various benchmarks in language modeling, commonsense reasoning, and long-context retrieval, while using only 58.9% of the original parameters. In this regard, HADES bridges GSP and neural sequence modeling, enabling efficient, hierarchical, and interpretable filtering within state-space models.