Just Fine-tune Twice: Selective Differential Privacy for Large Language Models

TL;DR

This paper introduces Just Fine-tune Twice (JFT), a novel framework for applying Selective Differential Privacy to large transformer models, enhancing privacy protection while maintaining model utility.

Contribution

The paper develops a new, easy-to-implement two-step fine-tuning method that extends SDP to transformer models and addresses imperfect policy implementation.

Findings

JFT achieves strong utility compared to baselines.

The method effectively protects sensitive tokens in large language models.

Empirical analysis confirms SDP privacy guarantees.

Abstract

Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying differential privacy (DP), a canonical notion with provable privacy guarantees for machine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP notion extension called Selective Differential Privacy (SDP) to protect only the sensitive tokens defined by a policy function. However, their algorithm only works for RNN-based models. In this paper, we develop a novel framework, Just Fine-tune Twice (JFT), that achieves SDP for state-of-the-art large transformer-based models. Our method is easy to implement: it first fine-tunes the model with redacted in-domain data, and then fine-tunes it again with the original in-domain data using a private training mechanism. Furthermore, we study the scenario of imperfect implementation of policy functions that misses sensitive tokens and develop systematic methods to handle it. Experiments show that our method achieves strong utility compared to previous baselines. We also analyze the SDP privacy guarantee empirically with the canary insertion attack.