Adapting to Linear Separable Subsets with Large-Margin in Differentially Private Learning

TL;DR

This paper introduces a differentially private algorithm for linear classification that adapts to data separability and outliers, providing improved risk bounds without prior knowledge of data margin or outlier subset.

Contribution

The paper presents a novel DP-ERM algorithm that adapts to large-margin separability and outliers, with improved bounds and no need for prior parameter knowledge.

Findings

Achieves an empirical risk bound of O(1/(\u03b3^2 \u03b5 n) + |S_out|/(b3 n))

Improves results in the presence of few outliers in the agnostic setting

Provides utility bounds for private hyperparameter tuning

Abstract

This paper studies the problem of differentially private empirical risk minimization (DP-ERM) for binary linear classification. We obtain an efficient $(\varepsilon,\delta)$-DP algorithm with an empirical zero-one risk bound of $\tilde{O}\left(\frac{1}{\gamma^2\varepsilon n} + \frac{|S_{\mathrm{out}}|}{\gamma n}\right)$ where $n$ is the number of data points, $S_{\mathrm{out}}$ is an arbitrary subset of data one can remove and $\gamma$ is the margin of linear separation of the remaining data points (after $S_{\mathrm{out}}$ is removed). Here, $\tilde{O}(\cdot)$ hides only logarithmic terms. In the agnostic case, we improve the existing results when the number of outliers is small. Our algorithm is highly adaptive because it does not require knowing the margin parameter $\gamma$ or outlier subset $S_{\mathrm{out}}$. We also derive a utility bound for the advanced private hyperparameter tuning algorithm.