Differentially Private Bayesian Optimization

TL;DR

This paper introduces differentially private methods for Bayesian optimization, enabling the private release of hyper-parameters and classifier accuracy while maintaining near-optimal performance guarantees under certain assumptions.

Contribution

It develops novel privacy-preserving techniques for Bayesian optimization that leverage differential privacy and convergence bounds, ensuring data privacy without sacrificing much utility.

Findings

Private hyper-parameters and accuracy can be released with strong guarantees.

Under Gaussian process assumptions, privacy-preserving outputs are near-optimal.

Methods remain effective even without GP assumptions using alternative smoothness guarantees.

Abstract

Bayesian optimization is a powerful tool for fine-tuning the hyper-parameters of a wide variety of machine learning models. The success of machine learning has led practitioners in diverse real-world settings to learn classifiers for practical problems. As machine learning becomes commonplace, Bayesian optimization becomes an attractive method for practitioners to automate the process of classifier hyper-parameter tuning. A key observation is that the data used for tuning models in these settings is often sensitive. Certain data such as genetic predisposition, personal email statistics, and car accident history, if not properly private, may be at risk of being inferred from Bayesian optimization outputs. To address this, we introduce methods for releasing the best hyper-parameters and classifier accuracy privately. Leveraging the strong theoretical guarantees of differential privacy and known Bayesian optimization convergence bounds, we prove that under a GP assumption these private quantities are also near-optimal. Finally, even if this assumption is not satisfied, we can use different smoothness guarantees to protect privacy.