High-Dimensional Private Linear Regression with Optimal Rates

TL;DR

This paper analyzes high-dimensional differentially private linear regression with gradient descent, establishing optimal risk bounds and demonstrating the benefits of gradient clipping and learning rate schedules.

Contribution

It provides a deterministic equivalent for DP-GD trajectories, analyzes the impact of gradient clipping, and proves minimax optimal risk bounds in high-dimensional settings.

Findings

DP-GD achieves minimax optimal risk of O(γ + γ^2/ρ^2) for well-conditioned data.

Gradient clipping smaller than typical gradient norm improves performance.

Risk exhibits power-law scaling in ill-conditioned data with spectral decay.

Abstract

Differentially private (DP) linear regression has received significant attention in the recent theoretical literature, with several approaches proposed to improve error rates. Our work considers the popular high-dimensional regime with random data, where the number of training samples $n$ and the input dimension $d$ grow at a proportional rate $d / n \to \gamma$, and it studies a family of one-pass DP gradient descent (DP-GD) algorithms satisfying $\rho^2 / 2$ zero concentrated DP. In this setting, we establish a deterministic equivalent for the DP-GD trajectory in terms of a system of ordinary differential equations. This allows to analyze the effect of gradient clipping constants that are smaller than the typical norm of the per-sample gradients - a setup shown to improve performance in practice. For well-conditioned data, we show that DP-GD, upon properly choosing clipping constant and learning rate, achieves the non-asymptotic risk of $O(\gamma + \gamma^2 / \rho^2)$, and we establish that this rate is minimax optimal. Then, we consider the ill-conditioned case where the data covariance spectrum follows a power-law distribution, and we show that the risk displays a power-like scaling law in $\gamma$, highlighting the change in the exponent as a function of the privacy parameter $\rho$. Overall, our analysis demonstrates the benefits of practical algorithmic design choices, including aggressive gradient clipping and decaying learning rate schedules.