FLASepformer: Efficient Speech Separation with Gated Focused Linear Attention Transformer

TL;DR

The paper introduces FLASepformer, a linear complexity transformer-based model for speech separation that achieves state-of-the-art performance with reduced memory and faster inference on long sequences.

Contribution

It proposes a novel linear attention mechanism and a gated module, enabling efficient and effective speech separation with lower resource consumption.

Findings

FLASepformer matches state-of-the-art performance.

Model speeds up inference by up to 2.29x.

Memory usage is reduced by up to 31.9%.

Abstract

Speech separation always faces the challenge of handling prolonged time sequences. Past methods try to reduce sequence lengths and use the Transformer to capture global information. However, due to the quadratic time complexity of the attention module, memory usage and inference time still increase significantly with longer segments. To tackle this, we introduce Focused Linear Attention and build FLASepformer with linear complexity for efficient speech separation. Inspired by SepReformer and TF-Locoformer, we have two variants: FLA-SepReformer and FLA-TFLocoformer. We also add a new Gated module to improve performance further. Experimental results on various datasets show that FLASepformer matches state-of-the-art performance with less memory consumption and faster inference. FLA-SepReformer-T/B/L increases speed by 2.29x, 1.91x, and 1.49x, with 15.8%, 20.9%, and 31.9% GPU memory usage, proving our model's effectiveness.