Federated Domain Adaptation for ASR with Full Self-Supervision

TL;DR

This paper introduces a federated learning approach for on-device speech recognition that leverages self-supervision and data augmentation to improve performance without ground-truth transcriptions, reducing costs and preserving privacy.

Contribution

It proposes a novel federated domain adaptation system for ASR using full self-supervision and a cost-effective training method with selective weight adaptation.

Findings

Achieved 12.9% relative WER reduction over out-of-domain baseline

Reduced training cost with self-restricted RNN-T loss and partial weight adaptation

Attained 70% of the performance gain of fully supervised centralized training

Abstract

Cross-device federated learning (FL) protects user privacy by collaboratively training a model on user devices, therefore eliminating the need for collecting, storing, and manually labeling user data. While important topics such as the FL training algorithm, non-IID-ness, and Differential Privacy have been well studied in the literature, this paper focuses on two challenges of practical importance for improving on-device ASR: the lack of ground-truth transcriptions and the scarcity of compute resource and network bandwidth on edge devices. First, we propose a FL system for on-device ASR domain adaptation with full self-supervision, which uses self-labeling together with data augmentation and filtering techniques. The system can improve a strong Emformer-Transducer based ASR model pretrained on out-of-domain data, using in-domain audio without any ground-truth transcriptions. Second, to reduce the training cost, we propose a self-restricted RNN Transducer (SR-RNN-T) loss, a variant of alignment-restricted RNN-T that uses Viterbi alignments from self-supervision. To further reduce the compute and network cost, we systematically explore adapting only a subset of weights in the Emformer-Transducer. Our best training recipe achieves a $12.9\%$ relative WER reduction over the strong out-of-domain baseline, which equals $70\%$ of the reduction achievable with full human supervision and centralized training.