Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis

TL;DR

This paper evaluates Mamba models across speech separation, recognition, and synthesis tasks, comparing their performance and efficiency to transformers, and finds Mamba's advantages depend on task duration and model specifics.

Contribution

It introduces Mamba-based models for three speech tasks and provides a comprehensive comparison with transformers in terms of performance and efficiency.

Findings

Mamba models achieve comparable or better performance than transformers.

Mamba models are more memory and speed efficient for longer speech segments.

Efficiency advantages of Mamba depend on speech duration and task complexity.

Abstract

It is too early to conclude that Mamba is a better alternative to transformers for speech before comparing Mamba with transformers in terms of both performance and efficiency in multiple speech-related tasks. To reach this conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet for speech separation, ConMamba for speech recognition, and VALL-M for speech synthesis. We compare them with transformers of similar sizes in performance, memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable or higher performance than their transformer counterparts: Sepformer, Conformer, and VALL-E. They are more efficient than transformers in memory and speed for speech longer than a threshold duration, inversely related to the resolution of a speech token. Mamba for separation is the most efficient, and Mamba for recognition is the least. Further, we show that Mamba is not more efficient than transformer for speech shorter than the threshold duration and performs worse in models that require joint modeling of text and speech, such as cross or masked attention of two inputs. Therefore, we argue that the superiority of Mamba or transformer depends on particular problems and models. Code available at https://github.com/xi-j/Mamba-TasNet and https://github.com/xi-j/Mamba-ASR.