AdaCliP: Adaptive Clipping for Private SGD

TL;DR

AdaCliP introduces an adaptive clipping method for differentially private SGD that reduces noise addition and improves model accuracy, advancing privacy-preserving machine learning techniques.

Contribution

It proposes a novel coordinate-wise adaptive clipping approach for DP-SGD, providing theoretical guarantees and empirical improvements over existing methods.

Findings

Reduces noise in DP-SGD compared to previous methods.

Produces models with higher accuracy under privacy constraints.

Theoretically guarantees adaptive clipping benefits.

Abstract

Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradients and add noise proportional to the sensitivity of the modified gradients. Under this framework, we propose AdaCliP, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive clipping of the gradient. We empirically demonstrate that AdaCliP reduces the amount of added noise and produces models with better accuracy.