R+R:Understanding Hyperparameter Effects in DP-SGD

TL;DR

This paper investigates how hyperparameters affect DP-SGD's performance, aiming to clarify their influence to improve privacy-utility trade-offs and facilitate private learning adoption.

Contribution

It conducts a comprehensive replication study synthesizing prior conjectures, testing their validity across datasets and models, and quantifies the importance of hyperparameter interactions.

Findings

Replicated the relationship between clipping threshold and learning rate.

Found inconsistent effects of batch size and epochs on performance.

Quantified the combined importance of certain hyperparameters.

Abstract

Research on the effects of essential hyperparameters of DP-SGD lacks consensus, verification, and replication. Contradictory and anecdotal statements on their influence make matters worse. While DP-SGD is the standard optimization algorithm for privacy-preserving machine learning, its adoption is still commonly challenged by low performance compared to non-private learning approaches. As proper hyperparameter settings can improve the privacy-utility trade-off, understanding the influence of the hyperparameters promises to simplify their optimization towards better performance, and likely foster acceptance of private learning. To shed more light on these influences, we conduct a replication study: We synthesize extant research on hyperparameter influences of DP-SGD into conjectures, conduct a dedicated factorial study to independently identify hyperparameter effects, and assess which conjectures can be replicated across multiple datasets, model architectures, and differential privacy budgets. While we cannot (consistently) replicate conjectures about the main and interaction effects of the batch size and the number of epochs, we were able to replicate the conjectured relationship between the clipping threshold and learning rate. Furthermore, we were able to quantify the significant importance of their combination compared to the other hyperparameters.