Production federated keyword spotting via distillation, filtering, and joint federated-centralized training

TL;DR

This paper presents a federated learning approach for keyword spotting that combines distillation, filtering, and joint training to improve model performance on user devices, addressing data domain gaps and unlabeled data.

Contribution

It introduces a novel federated training framework with confidence filtering and joint federated-centralized training for keyword spotting on mobile devices.

Findings

Significant improvements in offline quality metrics.

Enhanced user experience in live A/B tests.

Effective handling of unlabeled data through confidence filtering.

Abstract

We trained a keyword spotting model using federated learning on real user devices and observed significant improvements when the model was deployed for inference on phones. To compensate for data domains that are missing from on-device training caches, we employed joint federated-centralized training. And to learn in the absence of curated labels on-device, we formulated a confidence filtering strategy based on user-feedback signals for federated distillation. These techniques created models that significantly improved quality metrics in offline evaluations and user-experience metrics in live A/B experiments.