Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data

TL;DR

This paper introduces FEDENHANCE, an unsupervised federated learning method for speech enhancement from non-IID data, demonstrating competitive results and improved convergence through transfer learning and mixed training strategies.

Contribution

The paper presents FEDENHANCE, a novel unsupervised federated learning framework for speech enhancement with non-IID data, including a new dataset for source separation research.

Findings

Achieves competitive performance with IID training.

Transfer learning improves convergence and performance.

Combines supervised and unsupervised updates effectively.

Abstract

We propose FEDENHANCE, an unsupervised federated learning (FL) approach for speech enhancement and separation with non-IID distributed data across multiple clients. We simulate a real-world scenario where each client only has access to a few noisy recordings from a limited and disjoint number of speakers (hence non-IID). Each client trains their model in isolation using mixture invariant training while periodically providing updates to a central server. Our experiments show that our approach achieves competitive enhancement performance compared to IID training on a single device and that we can further facilitate the convergence speed and the overall performance using transfer learning on the server-side. Moreover, we show that we can effectively combine updates from clients trained locally with supervised and unsupervised losses. We also release a new dataset LibriFSD50K and its creation recipe in order to facilitate FL research for source separation problems.