Study of Lightweight Transformer Architectures for Single-Channel Speech Enhancement

TL;DR

This paper introduces a lightweight, transformer-based speech enhancement model that achieves state-of-the-art performance with significantly fewer parameters and computational requirements, suitable for edge devices.

Contribution

The paper proposes a novel streamlined FTF transformer architecture with adversarial training, reducing complexity while maintaining or improving performance over existing models.

Findings

LCT-GAN requires only 6% of DeepFilterNet2's parameters with similar performance.

LCT-GAN saves 9% parameters and 10% multiply-accumulate operations compared to CCFNet+(Lite).

LCT-GAN outperforms more complex baseline models on standard datasets.

Abstract

In speech enhancement, achieving state-of-the-art (SotA) performance while adhering to the computational constraints on edge devices remains a formidable challenge. Networks integrating stacked temporal and spectral modelling effectively leverage improved architectures such as transformers; however, they inevitably incur substantial computational complexity and model expansion. Through systematic ablation analysis on transformer-based temporal and spectral modelling, we demonstrate that the architecture employing streamlined Frequency-Time-Frequency (FTF) stacked transformers efficiently learns global dependencies within causal context, while avoiding considerable computational demands. Utilising discriminators in training further improves learning efficacy and enhancement without introducing additional complexity during inference. The proposed lightweight, causal, transformer-based architecture with adversarial training (LCT-GAN) yields SoTA performance on instrumental metrics among contemporary lightweight models, but with far less overhead. Compared to DeepFilterNet2, the LCT-GAN only requires 6% of the parameters, at similar complexity and performance. Against CCFNet+(Lite), LCT-GAN saves 9% in parameters and 10% in multiply-accumulate operations yet yielding improved performance. Further, the LCT-GAN even outperforms more complex, common baseline models on widely used test datasets.