Enabling On-Device Training of Speech Recognition Models with Federated Dropout

TL;DR

This paper introduces federated dropout to enable on-device training of speech recognition models, reducing communication costs and allowing dynamic model size adjustment while maintaining accuracy.

Contribution

It proposes a novel federated dropout method with layer-wise variation, improving on-device training efficiency and enabling smaller sub-models with low error rates.

Findings

Federated dropout effectively reduces communication and computation costs.

Layer-wise dropout rate variation improves model training and inference.

Smaller sub-models achieve low word error rates independently.

Abstract

Federated learning can be used to train machine learning models on the edge on local data that never leave devices, providing privacy by default. This presents a challenge pertaining to the communication and computation costs associated with clients' devices. These costs are strongly correlated with the size of the model being trained, and are significant for state-of-the-art automatic speech recognition models. We propose using federated dropout to reduce the size of client models while training a full-size model server-side. We provide empirical evidence of the effectiveness of federated dropout, and propose a novel approach to vary the dropout rate applied at each layer. Furthermore, we find that federated dropout enables a set of smaller sub-models within the larger model to independently have low word error rates, making it easier to dynamically adjust the size of the model deployed for inference.