Deep Learning with Data Privacy via Residual Perturbation

TL;DR

This paper introduces a novel residual perturbation method based on stochastic differential equations for privacy-preserving deep learning, which guarantees differential privacy and improves utility over existing methods.

Contribution

It proposes a residual perturbation technique that injects Gaussian noise into ResNet residuals, ensuring differential privacy and reducing generalization gap.

Findings

Guarantees differential privacy with residual perturbation.

Outperforms DPSGD in utility while maintaining privacy.

Efficiently reduces generalization gap in deep learning.

Abstract

Protecting data privacy in deep learning (DL) is of crucial importance. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many existing mechanisms achieve privacy at the cost of significant utility degradation and computational overhead. In this paper, we propose a stochastic differential equation-based residual perturbation for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap of DL. Empirically, we show that residual perturbation is computationally efficient and outperforms the state-of-the-art differentially private stochastic gradient descent (DPSGD) in utility maintenance without sacrificing membership privacy.