Towards Understanding Generalization in DP-GD: A Case Study in Training Two-Layer CNNs

TL;DR

This paper explores how differentially private gradient descent (DP-GD) can outperform standard gradient descent (GD) in training two-layer CNNs, achieving better generalization and privacy preservation under certain conditions.

Contribution

It provides a theoretical and empirical analysis showing DP-GD can yield superior generalization in specific neural network training scenarios compared to GD.

Findings

DP-GD can outperform GD in generalization under certain signal-to-noise ratios.

Small signal-to-noise ratios can cause GD to have poor test accuracy, while DP-GD maintains good accuracy.

Numerical simulations support the theoretical advantages of DP-GD in privacy-preserving training.

Abstract

Modern deep learning techniques focus on extracting intricate information from data to achieve accurate predictions. However, the training datasets may be crowdsourced and include sensitive information, such as personal contact details, financial data, and medical records. As a result, there is a growing emphasis on developing privacy-preserving training algorithms for neural networks that maintain good performance while preserving privacy. In this paper, we investigate the generalization and privacy performances of the differentially private gradient descent (DP-GD) algorithm, which is a private variant of the gradient descent (GD) by incorporating additional noise into the gradients during each iteration. Moreover, we identify a concrete learning task where DP-GD can achieve superior generalization performance compared to GD in training two-layer Huberized ReLU convolutional neural networks (CNNs). Specifically, we demonstrate that, under mild conditions, a small signal-to-noise ratio can result in GD producing training models with poor test accuracy, whereas DP-GD can yield training models with good test accuracy and privacy guarantees if the signal-to-noise ratio is not too small. This indicates that DP-GD has the potential to enhance model performance while ensuring privacy protection in certain learning tasks. Numerical simulations are further conducted to support our theoretical results.