Correlated Noise Provably Beats Independent Noise for Differentially Private Learning

TL;DR

This paper demonstrates that correlated noise in differentially private learning algorithms can provably outperform traditional independent noise, providing analytical bounds and practical validation for improved utility in private deep learning.

Contribution

It introduces a theoretical framework for correlated noise in DP learning, deriving bounds and an efficient method to optimize noise correlation, surpassing prior semi-definite programming approaches.

Findings

Correlated noise improves utility over independent noise in DP learning.

Analytical bounds are derived for linear regression and convex functions.

Experimental validation shows improved performance in private deep learning.

Abstract

Differentially private learning algorithms inject noise into the learning process. While the most common private learning algorithm, DP-SGD, adds independent Gaussian noise in each iteration, recent work on matrix factorization mechanisms has shown empirically that introducing correlations in the noise can greatly improve their utility. We characterize the asymptotic learning utility for any choice of the correlation function, giving precise analytical bounds for linear regression and as the solution to a convex program for general convex functions. We show, using these bounds, how correlated noise provably improves upon vanilla DP-SGD as a function of problem parameters such as the effective dimension and condition number. Moreover, our analytical expression for the near-optimal correlation function circumvents the cubic complexity of the semi-definite program used to optimize the noise correlation matrix in previous work. We validate our theory with experiments on private deep learning. Our work matches or outperforms prior work while being efficient both in terms of compute and memory.