Unlocking High-Accuracy Differentially Private Image Classification through Scale

TL;DR

This paper demonstrates that with proper hyper-parameter tuning and techniques, differentially private training can achieve high accuracy on image classification tasks, surpassing previous state-of-the-art results.

Contribution

The authors show that DP-SGD can perform significantly better on over-parameterized models with careful tuning, achieving new state-of-the-art results on CIFAR-10 and ImageNet.

Findings

Achieved 81.4% accuracy on CIFAR-10 with DP-SGD, surpassing previous SOTA of 71.7%.

Reached 83.8% top-1 accuracy on ImageNet under differential privacy.

Close to non-private SOTA accuracy with differential privacy, reducing the privacy-accuracy gap.

Abstract

Differential Privacy (DP) provides a formal privacy guarantee preventing adversaries with access to a machine learning model from extracting information about individual training points. Differentially Private Stochastic Gradient Descent (DP-SGD), the most popular DP training method for deep learning, realizes this protection by injecting noise during training. However previous works have found that DP-SGD often leads to a significant degradation in performance on standard image classification benchmarks. Furthermore, some authors have postulated that DP-SGD inherently performs poorly on large models, since the norm of the noise required to preserve privacy is proportional to the model dimension. In contrast, we demonstrate that DP-SGD on over-parameterized models can perform significantly better than previously thought. Combining careful hyper-parameter tuning with simple techniques to ensure signal propagation and improve the convergence rate, we obtain a new SOTA without extra data on CIFAR-10 of 81.4% under (8, 10^{-5})-DP using a 40-layer Wide-ResNet, improving over the previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a remarkable 83.8% top-1 accuracy on ImageNet under (0.5, 8*10^{-7})-DP. Additionally, we also achieve 86.7% top-1 accuracy under (8, 8 \cdot 10^{-7})-DP, which is just 4.3% below the current non-private SOTA for this task. We believe our results are a significant step towards closing the accuracy gap between private and non-private image classification.