Shifted Interpolation for Differential Privacy

TL;DR

This paper introduces shifted interpolated processes to improve privacy analysis in differentially private machine learning, providing tighter bounds and exact characterizations especially for strongly convex optimization.

Contribution

It develops a novel shifted interpolation technique that refines privacy amplification analysis and extends to various convex optimization settings, including the first exact analysis for strongly convex cases.

Findings

Tighter privacy bounds through shifted interpolation.

Extension of privacy analysis to strongly convex optimization.

Recovery and extension of the $f$-DP characterization of the exponential mechanism.

Abstract

Noisy gradient descent and its variants are the predominant algorithms for differentially private machine learning. It is a fundamental question to quantify their privacy leakage, yet tight characterizations remain open even in the foundational setting of convex losses. This paper improves over previous analyses by establishing (and refining) the "privacy amplification by iteration" phenomenon in the unifying framework of $f$-differential privacy--which tightly captures all aspects of the privacy loss and immediately implies tighter privacy accounting in other notions of differential privacy, e.g., $(\varepsilon,\delta)$-DP and R\'enyi DP. Our key technical insight is the construction of shifted interpolated processes that unravel the popular shifted-divergences argument, enabling generalizations beyond divergence-based relaxations of DP. Notably, this leads to the first exact privacy analysis in the foundational setting of strongly convex optimization. Our techniques extend to many settings: convex/strongly convex, constrained/unconstrained, full/cyclic/stochastic batches, and all combinations thereof. As an immediate corollary, we recover the $f$-DP characterization of the exponential mechanism for strongly convex optimization in Gopi et al. (2022), and moreover extend this result to more general settings.