Near-Optimal Private Linear Regression via Iterative Hessian Mixing

TL;DR

This paper introduces a new differentially private linear regression method called iterative Hessian mixing, which uses Gaussian sketches to improve accuracy over existing approaches like AdaSSP, supported by theoretical analysis and extensive experiments.

Contribution

We propose the iterative Hessian mixing algorithm for DP-OLS, combining Gaussian sketches with novel utility analysis, surpassing prior methods like AdaSSP in accuracy.

Findings

Iterative Hessian mixing outperforms AdaSSP in accuracy.

The new method provides better privacy-utility trade-offs.

Extensive experiments confirm the superiority of the approach.

Abstract

We study differentially private ordinary least squares (DP-OLS) with bounded data. The dominant approach, adaptive sufficient-statistics perturbation (AdaSSP), adds an adaptively chosen perturbation to the sufficient statistics, namely, the matrix $X^{\top}X$ and the vector $X^{\top}Y$, and is known to achieve near-optimal accuracy and to have strong empirical performance. In contrast, methods that rely on Gaussian-sketching, which ensure differential privacy by pre-multiplying the data with a random Gaussian matrix, are widely used in federated and distributed regression, yet remain relatively uncommon for DP-OLS. In this work, we introduce the iterative Hessian mixing, a novel DP-OLS algorithm that relies on Gaussian sketches and is inspired by the iterative Hessian sketch algorithm. We provide utility analysis for the iterative Hessian mixing as well as a new analysis for the previous methods that rely on Gaussian sketches. Then, we show that our new approach circumvents the intrinsic limitations of the prior methods and provides non-trivial improvements over AdaSSP. We conclude by running an extensive set of experiments across standard benchmarks to demonstrate further that our approach consistently outperforms these prior baselines.