DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning

TL;DR

DP-MicroAdam is a novel memory-efficient adaptive optimizer for differentially private training, achieving faster convergence and better accuracy than existing methods across various benchmarks.

Contribution

We introduce DP-MicroAdam, the first adaptive DP optimizer with proven convergence and superior empirical performance, reducing compute and tuning requirements.

Findings

Outperforms existing adaptive DP optimizers.

Achieves competitive or better accuracy than DP-SGD.

Proven convergence at the optimal rate.

Abstract

Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We propose DP-MicroAdam, a memory-efficient and sparsity-aware adaptive DP optimizer. We prove that DP-MicroAdam converges in stochastic non-convex optimization at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, up to privacy-dependent constants. Empirically, DP-MicroAdam outperforms existing adaptive DP optimizers and achieves competitive or superior accuracy compared to DP-SGD across a range of benchmarks, including CIFAR-10, large-scale ImageNet training, and private fine-tuning of pretrained transformers. These results demonstrate that adaptive optimization can improve both performance and stability under differential privacy.