Linear-Time User-Level DP-SCO via Robust Statistics

TL;DR

This paper presents a linear-time, robust statistics-based algorithm for user-level differentially private stochastic convex optimization, improving privacy-utility trade-offs and computational efficiency over existing methods.

Contribution

It introduces a novel algorithm that uses robust statistics to bound sensitivity of SGD iterates, reducing noise and enhancing privacy-utility trade-off, with proven optimality up to logarithmic factors.

Findings

Achieves improved privacy-utility trade-off compared to DP-SGD.

Maintains computational efficiency with linear-time complexity.

Provides an information-theoretic lower bound confirming near-optimality.

Abstract

User-level differentially private stochastic convex optimization (DP-SCO) has garnered significant attention due to the paramount importance of safeguarding user privacy in modern large-scale machine learning applications. Current methods, such as those based on differentially private stochastic gradient descent (DP-SGD), often struggle with high noise accumulation and suboptimal utility due to the need to privatize every intermediate iterate. In this work, we introduce a novel linear-time algorithm that leverages robust statistics, specifically the median and trimmed mean, to overcome these challenges. Our approach uniquely bounds the sensitivity of all intermediate iterates of SGD with gradient estimation based on robust statistics, thereby significantly reducing the gradient estimation noise for privacy purposes and enhancing the privacy-utility trade-off. By sidestepping the repeated privatization required by previous methods, our algorithm not only achieves an improved theoretical privacy-utility trade-off but also maintains computational efficiency. We complement our algorithm with an information-theoretic lower bound, showing that our upper bound is optimal up to logarithmic factors and the dependence on $\epsilon$. This work sets the stage for more robust and efficient privacy-preserving techniques in machine learning, with implications for future research and application in the field.