Differentially Private Language Models Benefit from Public Pre-training

TL;DR

This paper demonstrates that fine-tuning publicly pre-trained language models with differential privacy techniques enhances their performance on private data, balancing privacy guarantees with high-quality language modeling.

Contribution

It introduces a method of DP fine-tuning that leverages public pre-training to improve private domain language model quality.

Findings

DP fine-tuning significantly improves private data performance

Public pre-training mitigates quality loss due to privacy constraints

High-quality private language models become feasible with this approach

Abstract

Language modeling is a keystone task in natural language processing. When training a language model on sensitive information, differential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving by tuning a public base model on a private corpus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.