Differentially Private Regression with Unbounded Covariates

TL;DR

This paper develops efficient differentially private algorithms for classical regression tasks with unbounded covariates, extending prior work to the sub-Gaussian regime and covering models like logistic regression and SVMs.

Contribution

It introduces novel differentially private algorithms for regression with unbounded covariates, expanding the scope beyond bounded covariate assumptions.

Findings

Algorithms work efficiently under Gaussian marginals.

Provides unbiased estimates for logistic regression and SVMs.

Extends differential privacy techniques to sub-Gaussian data.

Abstract

We provide computationally efficient, differentially private algorithms for the classical regression settings of Least Squares Fitting, Binary Regression and Linear Regression with unbounded covariates. Prior to our work, privacy constraints in such regression settings were studied under strong a priori bounds on covariates. We consider the case of Gaussian marginals and extend recent differentially private techniques on mean and covariance estimation (Kamath et al., 2019; Karwa and Vadhan, 2018) to the sub-gaussian regime. We provide a novel technical analysis yielding differentially private algorithms for the above classical regression settings. Through the case of Binary Regression, we capture the fundamental and widely-studied models of logistic regression and linearly-separable SVMs, learning an unbiased estimate of the true regression vector, up to a scaling factor.