Differentially Private Empirical Risk Minimization Revisited: Faster and More General

TL;DR

This paper advances differentially private ERM algorithms by achieving faster convergence and broader applicability, including non-convex functions, with improved utility bounds and reduced gradient complexity.

Contribution

It introduces new algorithms for differentially private ERM that are faster and more general, covering convex, non-convex, and high-dimensional settings with improved theoretical guarantees.

Findings

Achieves optimal or near-optimal utility bounds with less gradient complexity.

Extends privacy-preserving ERM to non-convex functions satisfying Polyak-Lojasiewicz condition.

Provides algorithms suitable for high-dimensional ($p \,\gg\, n$) convex loss functions.

Abstract

In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared with previous work. For ERM with smooth convex loss function in high-dimensional ($p\gg n$) setting, we give an algorithm which achieves the upper bound with less gradient complexity than previous ones. At last, we generalize the expected excess empirical risk from convex loss functions to non-convex ones satisfying the Polyak-Lojasiewicz condition and give a tighter upper bound on the utility than the one in \cite{ijcai2017-548}.