Towards Lightweight Adaptation of Speech Enhancement Models in Real-World Environments

TL;DR

This paper introduces a lightweight, self-supervised adaptation framework for speech enhancement models that significantly improves robustness in diverse real-world noise environments with minimal computational overhead.

Contribution

The proposed method uses low-rank adapters to enable efficient on-device model adaptation without retraining the entire network, outperforming existing approaches in real-world scenarios.

Findings

Achieves 1.51 dB SI-SDR improvement with less than 1% parameter updates.

Demonstrates stable convergence and improved perceptual quality.

Effective across 111 environments and 37 noise types.

Abstract

Recent studies have shown that post-deployment adaptation can improve the robustness of speech enhancement models in unseen noise conditions. However, existing methods often incur prohibitive computational and memory costs, limiting their suitability for on-device deployment. In this work, we investigate model adaptation in realistic settings with dynamic acoustic scene changes and propose a lightweight framework that augments a frozen backbone with low-rank adapters updated via self-supervised training. Experiments on sequential scene evaluations spanning 111 environments across 37 noise types and three signal-to-noise ratio ranges, including the challenging [-8, 0] dB range, show that our method updates fewer than 1% of the base model's parameters while achieving an average 1.51 dB SI-SDR improvement within only 20 updates per scene. Compared to state-of-the-art approaches, our framework achieves competitive or superior perceptual quality with smoother and more stable convergence, demonstrating its practicality for lightweight on-device adaptation of speech enhancement models under real-world acoustic conditions.