Differentially Private Distributed Learning for Language Modeling Tasks

TL;DR

This paper introduces a novel differentially private distributed fine-tuning method for language models that enhances prediction accuracy on user data, reduces communication costs, and maintains strong privacy guarantees.

Contribution

The paper presents a new technique for privacy-preserving distributed language model fine-tuning that outperforms existing methods in accuracy and efficiency.

Findings

Achieves 70% perplexity reduction on user language data

Improves keystroke saving rate by 8.7 percentage points

Provides a framework for evaluating differential privacy in distributed training

Abstract

One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirements of maintaining the quality on the general data and minimization of communication costs. We propose a novel technique that significantly improves prediction quality on users' language compared to a general model and outperforms gradient compression methods in terms of communication efficiency. The proposed procedure is fast and leads to an almost 70% perplexity reduction and 8.7 percentage point improvement in keystroke saving rate on informal English texts. We also show that the range of tasks our approach is applicable to is not limited by language modeling only. Finally, we propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees.