Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models

TL;DR

Speech-Mamba introduces a novel long-context speech recognition model that combines selective state space models with Transformers, enabling efficient long-range dependency modeling with near-linear scaling.

Contribution

It pioneers the integration of selective state space models into speech recognition, enhancing long-sequence modeling capabilities beyond previous Transformer limitations.

Findings

Outperforms traditional models on long speech sequences

Scales near-linearly with sequence length

Improves long-range dependency modeling in speech recognition

Abstract

Current automatic speech recognition systems struggle with modeling long speech sequences due to high quadratic complexity of Transformer-based models. Selective state space models such as Mamba has performed well on long-sequence modeling in natural language processing and computer vision tasks. However, research endeavors in speech technology tasks has been under-explored. We propose Speech-Mamba, which incorporates selective state space modeling in Transformer neural architectures. Long sequence representations with selective state space models in Speech-Mamba is complemented with lower-level representations from Transformer-based modeling. Speech-mamba achieves better capacity to model long-range dependencies, as it scales near-linearly with sequence length.