Mamba in Speech: Towards an Alternative to Self-Attention

TL;DR

This paper investigates the application of Mamba, an alternative to self-attention, in speech processing tasks, demonstrating that bidirectional Mamba improves performance in speech recognition and enhancement over vanilla Mamba.

Contribution

It introduces the use of bidirectional Mamba in speech tasks and shows its advantages as an alternative to self-attention in Transformer models.

Findings

BiMamba outperforms vanilla Mamba in speech tasks.

Bidirectional design enhances speech processing performance.

BiMamba is effective as a self-attention substitute in Transformers.

Abstract

Transformer and its derivatives have achieved success in diverse tasks across computer vision, natural language processing, and speech processing. To reduce the complexity of computations within the multi-head self-attention mechanism in Transformer, Selective State Space Models (i.e., Mamba) were proposed as an alternative. Mamba exhibited its effectiveness in natural language processing and computer vision tasks, but its superiority has rarely been investigated in speech signal processing. This paper explores solutions for applying Mamba to speech processing by discussing two typical speech processing tasks: speech recognition, which requires semantic and sequential information, and speech enhancement, which focuses primarily on sequential patterns. The experimental results confirm that bidirectional Mamba (BiMamba) consistently outperforms vanilla Mamba, highlighting the advantages of a bidirectional design for speech processing. Moreover, experiments demonstrate the effectiveness of BiMamba as an alternative to the self-attention module in the Transformer model and its derivates, particularly for the semantic-aware task. The crucial technologies for transferring Mamba to speech are then summarized in ablation studies and the discussion section, offering insights for extending this research to a broader scope of tasks.