The Power of Sampling: Dimension-free Risk Bounds in Private ERM

TL;DR

This paper introduces a sampling-based approach for private empirical risk minimization that achieves dimension-free risk bounds, effectively handling non-smooth convex objectives with low-rank gradients using only zeroth order oracles.

Contribution

It presents a novel sampling method combining the exponential mechanism to overcome high-dimensional challenges in private ERM, achieving rank-dependent risk bounds under low-rank gradient assumptions.

Findings

Achieves dimension-free risk bounds in private ERM.

Handles non-smooth convex objectives with low-rank gradients.

Establishes lower bounds showing no separation between constrained and unconstrained settings when gradients are full-rank.

Abstract

Differentially private empirical risk minimization (DP-ERM) is a fundamental problem in private optimization. While the theory of DP-ERM is well-studied, as large-scale models become prevalent, traditional DP-ERM methods face new challenges, including (1) the prohibitive dependence on the ambient dimension, (2) the highly non-smooth objective functions, (3) costly first-order gradient oracles. Such challenges demand rethinking existing DP-ERM methodologies. In this work, we show that the regularized exponential mechanism combined with existing samplers can address these challenges altogether: under the standard unconstrained domain and low-rank gradients assumptions, our algorithm can achieve rank-dependent risk bounds for non-smooth convex objectives using only zeroth order oracles, which was not accomplished by prior methods. This highlights the power of sampling in differential privacy. We further construct lower bounds, demonstrating that when gradients are full-rank, there is no separation between the constrained and unconstrained settings. Our lower bound is derived from a general black-box reduction from unconstrained to the constrained domain and an improved lower bound in the constrained setting, which might be of independent interest.