Efficient Hyperparameter Optimization for Differentially Private Deep Learning

TL;DR

This paper introduces a novel optimization framework for hyperparameter tuning in differentially private deep learning, proposing three efficient algorithms that outperform grid search on standard datasets.

Contribution

It formulates hyperparameter tuning in DPSGD as an optimization problem and systematically studies evolutionary, Bayesian, and reinforcement learning algorithms for this task.

Findings

Algorithms outperform grid search baseline

Effective hyperparameter tuning improves privacy-utility tradeoff

Open-source code available for reproducibility

Abstract

Tuning the hyperparameters in the differentially private stochastic gradient descent (DPSGD) is a fundamental challenge. Unlike the typical SGD, private datasets cannot be used many times for hyperparameter search in DPSGD; e.g., via a grid search. Therefore, there is an essential need for algorithms that, within a given search space, can find near-optimal hyperparameters for the best achievable privacy-utility tradeoffs efficiently. We formulate this problem into a general optimization framework for establishing a desirable privacy-utility tradeoff, and systematically study three cost-effective algorithms for being used in the proposed framework: evolutionary, Bayesian, and reinforcement learning. Our experiments, for hyperparameter tuning in DPSGD conducted on MNIST and CIFAR-10 datasets, show that these three algorithms significantly outperform the widely used grid search baseline. As this paper offers a first-of-a-kind framework for hyperparameter tuning in DPSGD, we discuss existing challenges and open directions for future studies. As we believe our work has implications to be utilized in the pipeline of private deep learning, we open-source our code at https://github.com/AmanPriyanshu/DP-HyperparamTuning.