Non-Asymptotic Analysis of Online Local Private Learning with SGD

TL;DR

This paper provides the first non-asymptotic convergence analysis for online local differential privacy (LDP) in stochastic gradient descent, offering practical insights into privacy-utility trade-offs in real-time private learning.

Contribution

It introduces a novel non-asymptotic analysis framework for DP-SGD in online LDP settings, bridging a significant gap in privacy-preserving optimization research.

Findings

Non-asymptotic bounds for private estimators in online LDP.

Guidelines on hyperparameter effects on convergence rates.

Validation through theoretical derivations and numerical experiments.

Abstract

Differentially Private Stochastic Gradient Descent (DP-SGD) has been widely used for solving optimization problems with privacy guarantees in machine learning and statistics. Despite this, a systematic non-asymptotic convergence analysis for DP-SGD, particularly in the context of online problems and local differential privacy (LDP) models, remains largely elusive. Existing non-asymptotic analyses have focused on non-private optimization methods, and hence are not applicable to privacy-preserving optimization problems. This work initiates the analysis to bridge this gap and opens the door to non-asymptotic convergence analysis of private optimization problems. A general framework is investigated for the online LDP model in stochastic optimization problems. We assume that sensitive information from individuals is collected sequentially and aim to estimate, in real-time, a static parameter that pertains to the population of interest. Most importantly, we conduct a comprehensive non-asymptotic convergence analysis of the proposed estimators in finite-sample situations, which gives their users practical guidelines regarding the effect of various hyperparameters, such as step size, parameter dimensions, and privacy budgets, on convergence rates. Our proposed estimators are validated in the theoretical and practical realms by rigorous mathematical derivations and carefully constructed numerical experiments.