DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)

TL;DR

This paper identifies a bias in DP-Adam caused by noise addition, proposes DP-AdamBC to correct it, and demonstrates significant accuracy improvements across various tasks.

Contribution

The paper introduces DP-AdamBC, a bias correction method for DP-Adam that restores its expected behavior and enhances performance in privacy-preserving deep learning.

Findings

DP-AdamBC improves accuracy by up to 3.5% in various tasks

Bias in DP-Adam's second moment estimator affects performance

DP-AdamBC restores Adam's expected behavior under differential privacy

Abstract

The Adam optimizer is a popular choice in contemporary deep learning, due to its strong empirical performance. However we observe that in privacy sensitive scenarios, the traditional use of Differential Privacy (DP) with the Adam optimizer leads to sub-optimal performance on several tasks. We find that this performance degradation is due to a DP bias in Adam's second moment estimator, introduced by the addition of independent noise in the gradient computation to enforce DP guarantees. This DP bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam. We propose DP-AdamBC, an optimization algorithm which removes the bias in the second moment estimation and retrieves the expected behaviour of Adam. Empirically, DP-AdamBC significantly improves the optimization performance of DP-Adam by up to 3.5% in final accuracy in image, text, and graph node classification tasks.