Federated Evaluation of On-device Personalization

TL;DR

This paper extends federated learning to evaluate personalization strategies for global models, demonstrating that personalized language models improve user experience on smartphones without compromising privacy.

Contribution

It introduces methods and tools for federated evaluation of personalization, providing insights into when personalization benefits users in large-scale mobile applications.

Findings

Personalization benefits a significant fraction of users.

Federated evaluation effectively assesses personalization strategies.

Language models improve with personalization in real-world settings.

Abstract

Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers. In this work, we describe methods to extend the federation framework to evaluate strategies for personalization of global models. We present tools to analyze the effects of personalization and evaluate conditions under which personalization yields desirable models. We report on our experiments personalizing a language model for a virtual keyboard for smartphones with a population of tens of millions of users. We show that a significant fraction of users benefit from personalization.