Efficient Private SCO for Heavy-Tailed Data via Averaged Clipping

TL;DR

This paper introduces a one-time clipping method for differentially private stochastic convex optimization on heavy-tailed data, achieving improved convergence and efficiency over prior multi-clipping approaches.

Contribution

It proposes a novel one-time clipping strategy with theoretical analysis, leading to better convergence bounds and efficiency for DP stochastic convex optimization on heavy-tailed data.

Findings

Achieves improved convergence bounds for DP stochastic convex optimization.

Demonstrates efficiency of one-time clipping over multi-clipping methods.

Validates theoretical results with numerical experiments.

Abstract

We consider stochastic convex optimization for heavy-tailed data with the guarantee of being differentially private (DP). Most prior works on differentially private stochastic convex optimization for heavy-tailed data are either restricted to gradient descent (GD) or performed multi-times clipping on stochastic gradient descent (SGD), which is inefficient for large-scale problems. In this paper, we consider a one-time clipping strategy and provide principled analyses of its bias and private mean estimation. We establish new convergence results and improved complexity bounds for the proposed algorithm called AClipped-dpSGD for constrained and unconstrained convex problems. We also extend our convergent analysis to the strongly convex case and non-smooth case (which works for generalized smooth objectives with H$\ddot{\text{o}}$lder-continuous gradients). All the above results are guaranteed with a high probability for heavy-tailed data. Numerical experiments are conducted to justify the theoretical improvement.