Improving the Convergence of Private Shuffled Gradient Methods with Public Data

TL;DR

This paper analyzes private shuffled gradient methods for convex ERM, revealing their limitations and proposing a hybrid approach with public data that improves empirical risk in differentially private training.

Contribution

It provides the first empirical excess risk bounds for DP-ShuffleG and introduces Interleaved-ShuffleG, a novel hybrid method leveraging public data to enhance privacy-utility trade-offs.

Findings

Data shuffling worsens empirical excess risk compared to DP-SGD.

Interleaved-ShuffleG reduces excess risk by integrating public data.

Experiments show Interleaved-ShuffleG outperforms baselines on multiple datasets.

Abstract

We consider the problem of differentially private (DP) convex empirical risk minimization (ERM). While the standard DP-SGD algorithm is theoretically well-established, practical implementations often rely on shuffled gradient methods that traverse the training data sequentially rather than sampling with replacement in each iteration. Despite their widespread use, the theoretical privacy-accuracy trade-offs of private shuffled gradient methods (\textit{DP-ShuffleG}) remain poorly understood, leading to a gap between theory and practice. In this work, we leverage privacy amplification by iteration (PABI) and a novel application of Stein's lemma to provide the first empirical excess risk bound of \textit{DP-ShuffleG}. Our result shows that data shuffling results in worse empirical excess risk for \textit{DP-ShuffleG} compared to DP-SGD. To address this limitation, we propose \textit{Interleaved-ShuffleG}, a hybrid approach that integrates public data samples in private optimization. By alternating optimization steps that use private and public samples, \textit{Interleaved-ShuffleG} effectively reduces empirical excess risk. Our analysis introduces a new optimization framework with surrogate objectives, varying levels of noise injection, and a dissimilarity metric, which can be of independent interest. Our experiments on diverse datasets and tasks demonstrate the superiority of \textit{Interleaved-ShuffleG} over several baselines.