Beyond Uniform Lipschitz Condition in Differentially Private Optimization

TL;DR

This paper extends the analysis of differentially private stochastic gradient descent by relaxing the uniform Lipschitz assumption to sample-dependent bounds, providing guidance on clip norm selection and new convergence results for heavy-tailed gradients.

Contribution

It introduces a generalized Lipschitz condition with sample-dependent bounds and offers practical guidance for clip norm tuning in DP-SGD, along with convergence analysis for unbounded, heavy-tailed gradients.

Findings

Guidance on choosing clip norm up to the minimum per-sample Lipschitz constant.

Experimental validation on 8 datasets demonstrating the effectiveness of the proposed method.

New convergence results for DP-SGD with heavy-tailed, unbounded Lipschitz constants.

Abstract

Most prior results on differentially private stochastic gradient descent (DP-SGD) are derived under the simplistic assumption of uniform Lipschitzness, i.e., the per-sample gradients are uniformly bounded. We generalize uniform Lipschitzness by assuming that the per-sample gradients have sample-dependent upper bounds, i.e., per-sample Lipschitz constants, which themselves may be unbounded. We provide principled guidance on choosing the clip norm in DP-SGD for convex over-parameterized settings satisfying our general version of Lipschitzness when the per-sample Lipschitz constants are bounded; specifically, we recommend tuning the clip norm only till values up to the minimum per-sample Lipschitz constant. This finds application in the private training of a softmax layer on top of a deep network pre-trained on public data. We verify the efficacy of our recommendation via experiments on 8 datasets. Furthermore, we provide new convergence results for DP-SGD on convex and nonconvex functions when the Lipschitz constants are unbounded but have bounded moments, i.e., they are heavy-tailed.