Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Steps

TL;DR

This paper introduces the first subquadratic algorithms for differentially private non-smooth empirical risk minimization and stochastic convex optimization, achieving near-optimal bounds with innovative smoothing and subsampling techniques.

Contribution

It presents the first subquadratic algorithms for private ERM and SCO in the non-smooth case, with optimal excess risk bounds and novel smoothing methods.

Findings

Achieves near-optimal excess empirical risk bounds.

Develops subquadratic gradient query algorithms for non-smooth functions.

Provides the first subquadratic algorithms for private ERM and SCO in the non-smooth setting.

Abstract

We study the differentially private Empirical Risk Minimization (ERM) and Stochastic Convex Optimization (SCO) problems for non-smooth convex functions. We get a (nearly) optimal bound on the excess empirical risk and excess population loss with subquadratic gradient complexity. More precisely, our differentially private algorithm requires $O(\frac{N^{3/2}}{d^{1/8}}+ \frac{N^2}{d})$ gradient queries for optimal excess empirical risk, which is achieved with the help of subsampling and smoothing the function via convolution. This is the first subquadratic algorithm for the non-smooth case when $d$ is super constant. As a direct application, using the iterative localization approach of Feldman et al. \cite{fkt20}, we achieve the optimal excess population loss for stochastic convex optimization problem, with $O(\min\{N^{5/4}d^{1/8},\frac{ N^{3/2}}{d^{1/8}}\})$ gradient queries. Our work makes progress towards resolving a question raised by Bassily et al. \cite{bfgt20}, giving first algorithms for private ERM and SCO with subquadratic steps. We note that independently Asi et al. \cite{afkt21} gave other algorithms for private ERM and SCO with subquadratic steps.