SepPrune: Structured Pruning for Efficient Deep Speech Separation

TL;DR

SepPrune introduces a structured pruning framework tailored for deep speech separation models, significantly reducing computational costs while maintaining high separation performance, enabling faster training and real-time application suitability.

Contribution

It is the first structured pruning method specifically designed for deep speech separation models, combining differentiable masking with channel pruning for efficiency.

Findings

Pruned models recover 85% of original performance with minimal fine-tuning.

Achieves 36× faster convergence compared to training from scratch.

Outperforms existing pruning methods in speech separation tasks.

Abstract

Although deep learning has substantially advanced speech separation in recent years, most existing studies continue to prioritize separation quality while overlooking computational efficiency, an essential factor for low-latency speech processing in real-time applications. In this paper, we propose SepPrune, the first structured pruning framework specifically designed to compress deep speech separation models and reduce their computational cost. SepPrune begins by analyzing the computational structure of a given model to identify layers with the highest computational burden. It then introduces a differentiable masking strategy to enable gradient-driven channel selection. Based on the learned masks, SepPrune prunes redundant channels and fine-tunes the remaining parameters to recover performance. Extensive experiments demonstrate that this learnable pruning paradigm yields substantial advantages for channel pruning in speech separation models, outperforming existing methods. Notably, a model pruned with SepPrune can recover 85% of the performance of a pre-trained model (trained over hundreds of epochs) with only one epoch of fine-tuning, and achieves convergence 36$\times$ faster than training from scratch. Code is available at https://github.com/itsnotacie/SepPrune.