GeoClip: Geometry-Aware Clipping for Differentially Private SGD

TL;DR

GeoClip introduces a geometry-aware gradient clipping method for DP-SGD that adaptively aligns with gradient distribution geometry, improving privacy-utility trade-offs without extra privacy cost.

Contribution

It proposes GeoClip, a novel framework that clips and perturbs gradients in a transformed basis aligned with their geometry, with convergence guarantees and optimal transformation derivation.

Findings

GeoClip outperforms existing adaptive clipping methods under the same privacy budget.

GeoClip effectively estimates the transformation using only noisy gradients.

Experiments on tabular and image datasets validate GeoClip's superior performance.

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. While recent adaptive methods improve performance by adjusting this threshold during training, they operate in the standard coordinate system and fail to account for correlations across the coordinates of the gradient. We propose GeoClip, a geometry-aware framework that clips and perturbs gradients in a transformed basis aligned with the geometry of the gradient distribution. GeoClip adaptively estimates this transformation using only previously released noisy gradients, incurring no additional privacy cost. We provide convergence guarantees for GeoClip and derive a closed-form solution for the optimal transformation that minimizes the amount of noise added while keeping the probability of gradient clipping under control. Experiments on both tabular and image datasets demonstrate that GeoClip consistently outperforms existing adaptive clipping methods under the same privacy budget.