DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning

TL;DR

This paper introduces DP-AdamW and its bias-corrected variant for private deep learning, demonstrating improved performance over existing DP optimizers through theoretical analysis and empirical evaluation across multiple tasks.

Contribution

It presents a new DP optimizer, DP-AdamW, with theoretical privacy and convergence guarantees, and empirically shows its superiority over prior DP optimizers.

Findings

DP-AdamW outperforms state-of-the-art DP optimizers in accuracy.

Incorporating bias correction in DP-AdamW reduces accuracy, contrary to expectations.

Empirical results span text, image, and graph classification tasks.

Abstract

As deep learning methods increasingly utilize sensitive data on a widespread scale, differential privacy (DP) offers formal guarantees to protect against information leakage during model training. A significant challenge remains in implementing DP optimizers that retain strong performance while preserving privacy. Recent advances introduced ever more efficient optimizers, with AdamW being a popular choice for training deep learning models because of strong empirical performance. We study \emph{DP-AdamW} and introduce \emph{DP-AdamW-BC}, a differentially private variant of the AdamW optimizer with DP bias correction for the second moment estimator. We start by showing theoretical results for privacy and convergence guarantees of DP-AdamW and DP-AdamW-BC. Then, we empirically analyze the behavior of both optimizers across multiple privacy budgets ($\epsilon = 1, 3, 7$). We find that DP-AdamW outperforms existing state-of-the-art differentially private optimizers like DP-SGD, DP-Adam, and DP-AdamBC, scoring over 15\% higher on text classification, up to 5\% higher on image classification, and consistently 1\% higher on graph node classification. Moreover, we empirically show that incorporating bias correction in DP-AdamW (DP-AdamW-BC) consistently decreases accuracy, in contrast to the improvement of DP-AdamBC improvement over DP-Adam.