Efficient Monaural Speech Enhancement using Spectrum Attention Fusion

TL;DR

This paper introduces Spectrum Attention Fusion, a novel approach that reduces the complexity of Transformer-based speech enhancement models while maintaining or improving performance, making them more efficient for practical use.

Contribution

The paper proposes Spectrum Attention Fusion, a method that replaces multiple self-attention layers with a convolutional module to reduce model size and computational cost.

Findings

Achieves comparable or better results than state-of-the-art models

Uses significantly fewer parameters (0.58M)

Maintains high speech enhancement quality

Abstract

Speech enhancement is a demanding task in automated speech processing pipelines, focusing on separating clean speech from noisy channels. Transformer based models have recently bested RNN and CNN models in speech enhancement, however at the same time they are much more computationally expensive and require much more high quality training data, which is always hard to come by. In this paper, we present an improvement for speech enhancement models that maintains the expressiveness of self-attention while significantly reducing model complexity, which we have termed Spectrum Attention Fusion. We carefully construct a convolutional module to replace several self-attention layers in a speech Transformer, allowing the model to more efficiently fuse spectral features. Our proposed model is able to achieve comparable or better results against SOTA models but with significantly smaller parameters (0.58M) on the Voice Bank + DEMAND dataset.