DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

TL;DR

DP-Forward introduces a novel method for privacy-preserving language model inference by directly perturbing embedding matrices during the forward pass, achieving strong privacy guarantees, improved utility, and reduced computational costs compared to traditional DP-SGD.

Contribution

It proposes DP-Forward, a new approach that applies differential privacy directly to embedding matrices in the forward pass of language models, with an analytic matrix Gaussian mechanism for minimal noise.

Findings

Utility nearly matches non-private baselines

Outperforms DP-SGD by up to 7.7 percentage points

Reduces time and memory costs by 3 times

Abstract

Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs). We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3$\times$ time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails.