An Iterative Algorithm for Differentially Private $k$-PCA with Adaptive Noise

TL;DR

This paper introduces a novel iterative algorithm for differentially private PCA that effectively estimates the top k eigenvectors with near-optimal utility, overcoming previous limitations related to sample size and noise.

Contribution

It presents the first algorithm capable of estimating the top k eigenvectors under differential privacy, with improved sample complexity and noise handling for arbitrary k.

Findings

Achieves near-optimal statistical error for k=1.

Provides a lower bound for k>1 matching the upper bound up to a factor of k.

Demonstrates experimental advantages over baseline methods.

Abstract

Given $n$ i.i.d. random matrices $A_i \in \mathbb{R}^{d \times d}$ that share a common expectation $\Sigma$, the objective of Differentially Private Stochastic PCA is to identify a subspace of dimension $k$ that captures the largest variance directions of $\Sigma$, while preserving differential privacy (DP) of each individual $A_i$. Existing methods either (i) require the sample size $n$ to scale super-linearly with dimension $d$, even under Gaussian assumptions on the $A_i$, or (ii) introduce excessive noise for DP even when the intrinsic randomness within $A_i$ is small. Liu et al. (2022a) addressed these issues for sub-Gaussian data but only for estimating the top eigenvector ($k=1$) using their algorithm DP-PCA. We propose the first algorithm capable of estimating the top $k$ eigenvectors for arbitrary $k \leq d$, whilst overcoming both limitations above. For $k=1$ our algorithm matches the utility guarantees of DP-PCA, achieving near-optimal statistical error even when $n = \tilde{\!O}(d)$. We further provide a lower bound for general $k > 1$, matching our upper bound up to a factor of $k$, and experimentally demonstrate the advantages of our algorithm over comparable baselines.