Insufficient Statistics Perturbation: Stable Estimators for Private Least Squares

TL;DR

This paper introduces a differentially private least squares algorithm that is both sample- and time-efficient, with error bounds independent of the condition number, improving over prior methods that required more data or had worse error dependence.

Contribution

The authors develop a new private least squares estimator with linear error dependence on dimension and independence from the condition number, using a novel stable estimator approach.

Findings

Achieves near-optimal accuracy for datasets with bounded leverage and residuals.

Requires fewer samples than previous private algorithms, with polynomial error dependence.

Operates efficiently in terms of computation time.

Abstract

We present a sample- and time-efficient differentially private algorithm for ordinary least squares, with error that depends linearly on the dimension and is independent of the condition number of $X^\top X$, where $X$ is the design matrix. All prior private algorithms for this task require either $d^{3/2}$ examples, error growing polynomially with the condition number, or exponential time. Our near-optimal accuracy guarantee holds for any dataset with bounded statistical leverage and bounded residuals. Technically, we build on the approach of Brown et al. (2023) for private mean estimation, adding scaled noise to a carefully designed stable nonprivate estimator of the empirical regression vector.