Empirical Risk Minimization in Non-interactive Local Differential Privacy: Efficiency and High Dimensional Case

TL;DR

This paper investigates empirical risk minimization under non-interactive local differential privacy, proposing efficient algorithms with low communication costs and analyzing error bounds in both low and high dimensional settings.

Contribution

It introduces polynomial approximation techniques to reduce sample complexity dependence on dimension and develops communication-efficient algorithms for ERM under local differential privacy.

Findings

Avoids exponential dependence on dimension in low-dimensional ERM

Proposes 1-bit communication algorithms with constant computation per player

Provides error bounds dependent on Gaussian width in high-dimensional cases

Abstract

In this paper, we study the Empirical Risk Minimization problem in the non-interactive local model of differential privacy. In the case of constant or low dimensionality ($p\ll n$), we first show that if the ERM loss function is $(\infty, T)$-smooth, then we can avoid a dependence of the sample complexity, to achieve error $\alpha$, on the exponential of the dimensionality $p$ with base $1/\alpha$ (i.e., $\alpha^{-p}$), which answers a question in [smith 2017 interaction]. Our approach is based on polynomial approximation. Then, we propose player-efficient algorithms with $1$-bit communication complexity and $O(1)$ computation cost for each player. The error bound is asymptotically the same as the original one. Also with additional assumptions we show a server efficient algorithm. Next we consider the high dimensional case ($n\ll p$), we show that if the loss function is Generalized Linear function and convex, then we could get an error bound which is dependent on the Gaussian width of the underlying constrained set instead of $p$, which is lower than that in [smith 2017 interaction].