Scaling Private Deep Learning with Low-Rank and Sparse Gradients

TL;DR

This paper introduces a unified framework called LSG that leverages low-rank and sparse structures in neural network gradients to improve the performance of differentially private training on large-scale models.

Contribution

The paper proposes a novel method combining low-rank approximation and gradient sparsification to reduce noise impact in DPSGD for large neural networks.

Findings

Outperforms state-of-the-art baselines in NLP and computer vision tasks.

Effectively reduces gradient dimension and noise in private training.

Maintains neural network performance with privacy guarantees.

Abstract

Applying Differentially Private Stochastic Gradient Descent (DPSGD) to training modern, large-scale neural networks such as transformer-based models is a challenging task, as the magnitude of noise added to the gradients at each iteration scales with model dimension, hindering the learning capability significantly. We propose a unified framework, $\textsf{LSG}$, that fully exploits the low-rank and sparse structure of neural networks to reduce the dimension of gradient updates, and hence alleviate the negative impacts of DPSGD. The gradient updates are first approximated with a pair of low-rank matrices. Then, a novel strategy is utilized to sparsify the gradients, resulting in low-dimensional, less noisy updates that are yet capable of retaining the performance of neural networks. Empirical evaluation on natural language processing and computer vision tasks shows that our method outperforms other state-of-the-art baselines.