Amplifying R\'enyi Differential Privacy via Shuffling

TL;DR

This paper analyzes the Re9nyi differential privacy of cyclic stochastic gradient descent, providing privacy guarantees comparable to sampling with replacement, with broad applicability due to assumption-free proof techniques.

Contribution

It extends Re9nyi differential privacy analysis to cyclic SGD, a faster alternative to traditional SGD, with general proof methods that do not depend on specific models or data.

Findings

Privacy guarantees for cyclic SGD are comparable to sampling with replacement.

Proof techniques are assumption-free and widely applicable.

Cyclic SGD maintains strong privacy guarantees in large-scale settings.

Abstract

Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the R\'enyi differential privacy of stochastic gradient descent when each training example is sampled without replacement (also known as cyclic SGD). Cyclic SGD is typically faster than traditional SGD and is the algorithm of choice in large-scale implementations. We recover privacy guarantees for cyclic SGD which are competitive with those known for sampling with replacement. Our proof techniques make no assumptions on the model or on the data and are hence widely applicable.