Privacy Amplification for BandMF via $b$-Min-Sep Subsampling

TL;DR

This paper introduces a new subsampling method called $b$-min-sep for privacy amplification in BandMF, improving privacy guarantees especially in low-noise regimes and extending to user-level privacy.

Contribution

The paper proposes $b$-min-sep subsampling, a novel scheme that generalizes existing methods and enhances privacy amplification for BandMF, with a detailed privacy analysis and practical extensions.

Findings

$b$-min-sep matches cyclic Poisson in high noise regimes.

$b$-min-sep achieves better privacy guarantees in mid-to-low noise regimes.

Experimental results support the improved privacy guarantees.

Abstract

We study privacy amplification for BandMF, i.e., DP-SGD with correlated noise across iterations via a banded correlation matrix. We propose $b$-min-sep subsampling, a new subsampling scheme that generalizes Poisson and balls-in-bins subsampling, extends prior practical batching strategies for BandMF, and enables stronger privacy amplification than cyclic Poisson while preserving the structural properties needed for analysis. We give a near-exact privacy analysis using Monte Carlo accounting, based on a dynamic program that leverages the Markovian structure in the subsampling procedure. We show that $b$-min-sep matches cyclic Poisson subsampling in the high noise regime and achieves strictly better guarantees in the mid-to-low noise regime, with experimental results that bolster our claims. We further show that unlike previous BandMF subsampling schemes, our $b$-min-sep subsampling naturally extends to the multi-attribution user-level privacy setting.