Differentially Private Ordinary Least Squares

TL;DR

This paper develops differentially private methods for linear regression analysis, enabling the construction of confidence intervals and hypothesis testing while preserving data privacy, using techniques like Johnson-Lindenstrauss Transform and Ridge regression.

Contribution

It introduces differentially private confidence intervals for linear regression using JLT and analyzes their effectiveness for well-spread data and regularized models.

Findings

JLT provides accurate approximation of t-values for well-spread data

Confidence intervals can be derived from projected data in Ridge regression

Analyze Gauss algorithm can produce valid confidence intervals under certain conditions

Abstract

Linear regression is one of the most prevalent techniques in machine learning, however, it is also common to use linear regression for its \emph{explanatory} capabilities rather than label prediction. Ordinary Least Squares (OLS) is often used in statistics to establish a correlation between an attribute (e.g. gender) and a label (e.g. income) in the presence of other (potentially correlated) features. OLS assumes a particular model that randomly generates the data, and derives \emph{$t$-values} --- representing the likelihood of each real value to be the true correlation. Using $t$-values, OLS can release a \emph{confidence interval}, which is an interval on the reals that is likely to contain the true correlation, and when this interval does not intersect the origin, we can \emph{reject the null hypothesis} as it is likely that the true correlation is non-zero. Our work aims at achieving similar guarantees on data under differentially private estimators. First, we show that for well-spread data, the Gaussian Johnson-Lindenstrauss Transform (JLT) gives a very good approximation of $t$-values, secondly, when JLT approximates Ridge regression (linear regression with $l_2$-regularization) we derive, under certain conditions, confidence intervals using the projected data, lastly, we derive, under different conditions, confidence intervals for the "Analyze Gauss" algorithm (Dwork et al, STOC 2014).