Beyond Covariance Matrix: The Statistical Complexity of Private Linear Regression

TL;DR

This paper investigates the statistical complexity of private linear regression, revealing that privacy constraints relate more to the L1-analogue of the covariance matrix than the usual covariance, and introduces optimal algorithms for privacy-preserving regression and bandits.

Contribution

It uncovers the importance of the L1-analogue of the covariance matrix in private regression and develops minimax optimal algorithms for both central and local privacy models.

Findings

Established minimax convergence rates for private linear regression.

Introduced an Information-Weighted Regression method achieving optimal rates.

Designed a private linear contextual bandit algorithm with rate-optimal regret bounds.

Abstract

We study the statistical complexity of private linear regression under an unknown, potentially ill-conditioned covariate distribution. Somewhat surprisingly, under privacy constraints the intrinsic complexity is \emph{not} captured by the usual covariance matrix but rather its $L_1$ analogues. Building on this insight, we establish minimax convergence rates for both the central and local privacy models and introduce an Information-Weighted Regression method that attains the optimal rates. As application, in private linear contextual bandits, we propose an efficient algorithm that achieves rate-optimal regret bounds of order $\sqrt{T}+\frac{1}{\alpha}$ and $\sqrt{T}/\alpha$ under joint and local $\alpha$-privacy models, respectively. Notably, our results demonstrate that joint privacy comes at almost no additional cost, addressing the open problems posed by Azize and Basu (2024).