Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

TL;DR

This paper develops new differentially private stochastic optimization algorithms that effectively handle heavy-tailed gradient data, achieving optimal convergence rates that match theoretical lower bounds.

Contribution

It introduces a simple clipping method and a more complex iterative approach for DP optimization with heavy-tailed gradients, achieving optimal rates.

Findings

Achieves optimal population risk bounds under heavy-tailed gradients.

Improves over existing methods by carefully handling tail behavior.

Matches the minimax lower bound, indicating optimality.

Abstract

We study convex optimization problems under differential privacy (DP). With heavy-tailed gradients, existing works achieve suboptimal rates. The main obstacle is that existing gradient estimators have suboptimal tail properties, resulting in a superfluous factor of $d$ in the union bound. In this paper, we explore algorithms achieving optimal rates of DP optimization with heavy-tailed gradients. Our first method is a simple clipping approach. Under bounded $p$-th order moments of gradients, with $n$ samples, it achieves $\tilde{O}(\sqrt{d/n}+\sqrt{d}(\sqrt{d}/n\epsilon)^{1-1/p})$ population risk with $\epsilon\leq 1/\sqrt{d}$. We then propose an iterative updating method, which is more complex but achieves this rate for all $\epsilon\leq 1$. The results significantly improve over existing methods. Such improvement relies on a careful treatment of the tail behavior of gradient estimators. Our results match the minimax lower bound in \cite{kamath2022improved}, indicating that the theoretical limit of stochastic convex optimization under DP is achievable.