Enhancing Differentially Private Linear Regression via Public Second-Moment

TL;DR

This paper introduces a new differentially private linear regression method that uses public second-moment information to improve accuracy and robustness, outperforming traditional approaches.

Contribution

It proposes a transformed sufficient statistics perturbation approach leveraging public second-moment data to enhance DP linear regression accuracy.

Findings

Improved condition number of the second-moment matrix.

Theoretical error bounds showing enhanced robustness.

Experimental results confirming better utility on real datasets.

Abstract

Leveraging information from public data has become increasingly crucial in enhancing the utility of differentially private (DP) methods. Traditional DP approaches often require adding noise based solely on private data, which can significantly degrade utility. In this paper, we address this limitation in the context of the ordinary least squares estimator (OLSE) of linear regression based on sufficient statistics perturbation (SSP) under the unbounded data assumption. We propose a novel method that involves transforming private data using the public second-moment matrix to compute a transformed SSP-OLSE, whose second-moment matrix yields a better condition number and improves the OLSE accuracy and robustness. We derive theoretical error bounds about our method and the standard SSP-OLSE to the non-DP OLSE, which reveal the improved robustness and accuracy achieved by our approach. Experiments on synthetic and real-world datasets demonstrate the utility and effectiveness of our method.