On the Convergence of DP-SGD with Adaptive Clipping

TL;DR

This paper analyzes the convergence of adaptive gradient clipping in differentially private SGD, providing the first theoretical guarantees for quantile clipping methods and guiding parameter choices for practical use.

Contribution

It offers the first comprehensive convergence analysis of quantile clipping in DP-SGD, addressing bias issues and establishing theoretical guarantees.

Findings

Quantile clipping can be biased similarly to constant threshold clipping.

Careful scheduling of quantile and step size mitigates bias and ensures convergence.

Theoretical guarantees are established for DP-QC-SGD, supporting practical adaptive clipping heuristics.

Abstract

Stochastic Gradient Descent (SGD) with gradient clipping is a powerful technique for enabling differentially private optimization. Although prior works extensively investigated clipping with a constant threshold, private training remains highly sensitive to threshold selection, which can be expensive or even infeasible to tune. This sensitivity motivates the development of adaptive approaches, such as quantile clipping, which have demonstrated empirical success but lack a solid theoretical understanding. This paper provides the first comprehensive convergence analysis of SGD with quantile clipping (QC-SGD). We demonstrate that QC-SGD suffers from a bias problem similar to constant-threshold clipped SGD but show how this can be mitigated through a carefully designed quantile and step size schedule. Our analysis reveals crucial relationships between quantile selection, step size, and convergence behavior, providing practical guidelines for parameter selection. We extend these results to differentially private optimization, establishing the first theoretical guarantees for DP-QC-SGD. Our findings provide theoretical foundations for widely used adaptive clipping heuristic and highlight open avenues for future research.