DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework

TL;DR

This paper introduces DP-HyPO, a novel framework for adaptive private hyperparameter optimization that enhances model performance while preserving data privacy, bridging the gap with non-private methods.

Contribution

The paper presents the first adaptive private hyperparameter optimization framework with differential privacy guarantees, addressing a key gap in privacy-preserving machine learning.

Findings

Effective on diverse real-world datasets

Provides differential privacy analysis

Outperforms random hyperparameter selection

Abstract

Hyperparameter optimization, also known as hyperparameter tuning, is a widely recognized technique for improving model performance. Regrettably, when training private ML models, many practitioners often overlook the privacy risks associated with hyperparameter optimization, which could potentially expose sensitive information about the underlying dataset. Currently, the sole existing approach to allow privacy-preserving hyperparameter optimization is to uniformly and randomly select hyperparameters for a number of runs, subsequently reporting the best-performing hyperparameter. In contrast, in non-private settings, practitioners commonly utilize ``adaptive'' hyperparameter optimization methods such as Gaussian process-based optimization, which select the next candidate based on information gathered from previous outputs. This substantial contrast between private and non-private hyperparameter optimization underscores a critical concern. In our paper, we introduce DP-HyPO, a pioneering framework for ``adaptive'' private hyperparameter optimization, aiming to bridge the gap between private and non-private hyperparameter optimization. To accomplish this, we provide a comprehensive differential privacy analysis of our framework. Furthermore, we empirically demonstrate the effectiveness of DP-HyPO on a diverse set of real-world datasets.