Deep Learning under Fractional-Order Differential Privacy

TL;DR

This paper introduces FO-DP-SGD, a fractional-memory extension of DP-SGD that improves privacy-utility trade-offs by incorporating long-memory effects into the privacy mechanism.

Contribution

It proposes a novel fractional recursive query mechanism that enhances privacy accounting and utility in differentially private stochastic gradient descent.

Findings

FO-DP-SGD achieves better test accuracy than DP-SGD and other private baselines.

The method maintains standard privacy guarantees with effective noise-to-sensitivity ratio.

Experiments on SVHN, CIFAR-10, and CIFAR-100 demonstrate improved privacy-utility performance.

Abstract

Differentially private stochastic gradient descent (DP-SGD) is a standard approach to privacy-preserving learning based on per-example clipping, subsampling, Gaussian perturbation, and privacy accounting. Classical DP-SGD releases a noisy version of the current clipped subsampled gradient sum. We propose Fractional-Order Differentially Private Stochastic Gradient Descent (\textbf{FO-DP-SGD}), a mechanism-level extension that replaces this current-only query, before Gaussian noise is added, with a fractional recursive query combining the current clipped sum with a finite-window, power-law-weighted aggregation of previously released private sum-level outputs. This injects fractional memory into the release mechanism while preserving the standard \emph{sum-then-noise-then-divide} structure. Under add/remove adjacency with Poisson subsampling, the current-step sensitivity analysis shows that the only newly data-dependent term is the scaled current clipped sum. Hence, conditioned on the private history, the effective \(\ell_2\)-sensitivity is at most \(\beta C\), where \(C\) is the clipping threshold and \(\beta\in(0,1]\) controls the current-step contribution. Thus, FO-DP-SGD admits standard per-step R\'enyi differential privacy accounting via a Poisson-subsampled Gaussian mechanism with effective noise-to-sensitivity ratio \(\sigma/\beta\), and composes to yield overall \((\varepsilon,\delta)\)-differential privacy guarantees. FO-DP-SGD provides a framework for studying long-memory effects in private optimization. The fractional order, memory window, and mixing coefficient govern the trade-off among current-step sensitivity, signal retention, and private-history influence. Experiments on SVHN, CIFAR-10, and CIFAR-100 show improved test accuracy and privacy--utility performance over DP-SGD and private baselines including DP-Adam, DP-IS, SA-DP-SGD, ADP-AdamW, DP-SAT, and DP-Adam-AC.