Momentum Aggregation for Private Non-convex ERM

TL;DR

This paper presents new algorithms with improved convergence guarantees for privacy-preserving non-convex ERM, utilizing advanced sensitivity analysis and momentum techniques to achieve better gradient bounds under differential privacy constraints.

Contribution

It introduces a novel sensitivity analysis for stochastic gradient descent on smooth objectives combined with momentum and private aggregation, leading to improved gradient bounds in private non-convex ERM.

Findings

Achieves a gradient norm of O(d^{1/3}/(\u03b5 N)^{2/3}) with differential privacy.

Provides convergence guarantees with fewer gradient evaluations than previous methods.

Improves the theoretical understanding of private non-convex optimization algorithms.

Abstract

We introduce new algorithms and convergence guarantees for privacy-preserving non-convex Empirical Risk Minimization (ERM) on smooth $d$-dimensional objectives. We develop an improved sensitivity analysis of stochastic gradient descent on smooth objectives that exploits the recurrence of examples in different epochs. By combining this new approach with recent analysis of momentum with private aggregation techniques, we provide an $(\epsilon,\delta)$-differential private algorithm that finds a gradient of norm $\tilde O\left(\frac{d^{1/3}}{(\epsilon N)^{2/3}}\right)$ in $O\left(\frac{N^{7/3}\epsilon^{4/3}}{d^{2/3}}\right)$ gradient evaluations, improving the previous best gradient bound of $\tilde O\left(\frac{d^{1/4}}{\sqrt{\epsilon N}}\right)$.