Differentially Private Stochastic Gradient Descent with Low-Noise

TL;DR

This paper develops and analyzes differentially private SGD algorithms that achieve optimal excess risk bounds in low-noise convex and pairwise learning settings, balancing privacy and utility.

Contribution

It introduces a new differentially private SGD algorithm with sharper utility bounds and establishes the first fast learning rates for privacy-preserving pairwise learning under low-noise conditions.

Findings

Sharper excess risk bounds for private SGD in low-noise convex optimization

Proposed a simple private SGD for pairwise learning with optimal rates

Established the first fast learning rates for privacy-preserving pairwise learning

Abstract

Modern machine learning algorithms aim to extract fine-grained information from data to provide accurate predictions, which often conflicts with the goal of privacy protection. This paper addresses the practical and theoretical importance of developing privacy-preserving machine learning algorithms that ensure good performance while preserving privacy. In this paper, we focus on the privacy and utility (measured by excess risk bounds) performances of differentially private stochastic gradient descent (SGD) algorithms in the setting of stochastic convex optimization. Specifically, we examine the pointwise problem in the low-noise setting for which we derive sharper excess risk bounds for the differentially private SGD algorithm. In the pairwise learning setting, we propose a simple differentially private SGD algorithm based on gradient perturbation. Furthermore, we develop novel utility bounds for the proposed algorithm, proving that it achieves optimal excess risk rates even for non-smooth losses. Notably, we establish fast learning rates for privacy-preserving pairwise learning under the low-noise condition, which is the first of its kind.