Exploring Machine Learning Privacy/Utility trade-off from a hyperparameters Lens

TL;DR

This paper investigates how hyperparameters influence the privacy and utility of differentially private ML models, revealing that certain choices like Bounded RELU can enhance utility without compromising privacy.

Contribution

It systematically explores the impact of hyperparameters on privacy-preserving models and achieves state-of-the-art accuracy on multiple datasets using simple modifications.

Findings

Bounded RELU improves utility while maintaining privacy

Achieved new state-of-the-art accuracy on MNIST, FashionMNIST, CIFAR-10

Hyperparameters significantly affect privacy-utility trade-offs

Abstract

Machine Learning (ML) architectures have been applied to several applications that involve sensitive data, where a guarantee of users' data privacy is required. Differentially Private Stochastic Gradient Descent (DPSGD) is the state-of-the-art method to train privacy-preserving models. However, DPSGD comes at a considerable accuracy loss leading to sub-optimal privacy/utility trade-offs. Towards investigating new ground for better privacy-utility trade-off, this work questions; (i) if models' hyperparameters have any inherent impact on ML models' privacy-preserving properties, and (ii) if models' hyperparameters have any impact on the privacy/utility trade-off of differentially private models. We propose a comprehensive design space exploration of different hyperparameters such as the choice of activation functions, the learning rate and the use of batch normalization. Interestingly, we found that utility can be improved by using Bounded RELU as activation functions with the same privacy-preserving characteristics. With a drop-in replacement of the activation function, we achieve new state-of-the-art accuracy on MNIST (96.02\%), FashionMnist (84.76\%), and CIFAR-10 (44.42\%) without any modification of the learning procedure fundamentals of DPSGD.