Federated Learning Of Out-Of-Vocabulary Words

TL;DR

This paper shows that federated learning enables a character-level RNN to learn out-of-vocabulary words on user devices, improving vocabulary expansion for smartphones without compromising user privacy.

Contribution

It introduces a federated learning approach for training character-level RNNs to learn OOV words directly on user devices, preserving privacy and expanding vocabulary.

Findings

High recall and precision in simulated federated learning setting

Effective learning of meaningful OOV words on mobile devices

Demonstrated practicality of privacy-preserving vocabulary expansion

Abstract

We demonstrate that a character-level recurrent neural network is able to learn out-of-vocabulary (OOV) words under federated learning settings, for the purpose of expanding the vocabulary of a virtual keyboard for smartphones without exporting sensitive text to servers. High-frequency words can be sampled from the trained generative model by drawing from the joint posterior directly. We study the feasibility of the approach in two settings: (1) using simulated federated learning on a publicly available non-IID per-user dataset from a popular social networking website, (2) using federated learning on data hosted on user mobile devices. The model achieves good recall and precision compared to ground-truth OOV words in setting (1). With (2) we demonstrate the practicality of this approach by showing that we can learn meaningful OOV words with good character-level prediction accuracy and cross entropy loss.