Privacy-Accuracy Trade-offs in High-Dimensional LASSO under Perturbation Mechanisms

TL;DR

This paper analyzes the privacy-accuracy trade-offs in high-dimensional LASSO regression under differential privacy mechanisms, using AMP to characterize estimator behavior and revealing how sparsity influences privacy stability.

Contribution

It introduces an AMP-based analysis of privacy-preserving LASSO, comparing output and objective perturbation mechanisms and highlighting the role of sparsity in privacy-accuracy trade-offs.

Findings

Sparsity improves privacy by stabilizing the estimator.

Objective perturbation can have non-monotonic effects with increasing noise.

AMP effectively characterizes privacy-accuracy trade-offs in high-dimensional models.

Abstract

We study privacy-preserving sparse linear regression in the high-dimensional regime, focusing on the LASSO estimator. We analyze two widely used mechanisms for differential privacy: output perturbation, which injects noise into the estimator, and objective perturbation, which adds a random linear term to the loss function. Using approximate message passing (AMP), we characterize the typical behavior of these estimators under random design and privacy noise. To quantify privacy, we adopt typical-case measures, including the on-average KL divergence, which admits a hypothesis-testing interpretation in terms of distinguishability between neighboring datasets. Our analysis reveals that sparsity plays a central role in shaping the privacy-accuracy trade-off: stronger regularization can improve privacy by stabilizing the estimator against single-point data changes. We further show that the two mechanisms exhibit qualitatively different behaviors. In particular, for objective perturbation, increasing the noise level can have non-monotonic effects, and excessive noise may destabilize the estimator, leading to increased sensitivity to data perturbations. Our results demonstrate that AMP provides a powerful framework for analyzing privacy-accuracy trade-offs in high-dimensional sparse models.