On the utility and protection of optimization with differential privacy and classic regularization techniques

TL;DR

This paper compares differential privacy with standard regularization techniques in deep learning, analyzing their impact on model utility, training, and privacy against attacks, highlighting the limitations of differential privacy.

Contribution

It provides an empirical comparison showing that regularization techniques can sometimes offer better privacy and utility trade-offs than differential privacy methods.

Findings

Dropout and L2-regularization often outperform DP-SGD in privacy preservation.

Regularization techniques can maintain higher model utility under privacy constraints.

Differential privacy has notable flaws and limitations in practical scenarios.

Abstract

Nowadays, owners and developers of deep learning models must consider stringent privacy-preservation rules of their training data, usually crowd-sourced and retaining sensitive information. The most widely adopted method to enforce privacy guarantees of a deep learning model nowadays relies on optimization techniques enforcing differential privacy. According to the literature, this approach has proven to be a successful defence against several models' privacy attacks, but its downside is a substantial degradation of the models' performance. In this work, we compare the effectiveness of the differentially-private stochastic gradient descent (DP-SGD) algorithm against standard optimization practices with regularization techniques. We analyze the resulting models' utility, training performance, and the effectiveness of membership inference and model inversion attacks against the learned models. Finally, we discuss differential privacy's flaws and limits and empirically demonstrate the often superior privacy-preserving properties of dropout and l2-regularization.