Federated PCA and Estimation for Spiked Covariance Matrices: Optimal Rates and Efficient Algorithm

TL;DR

This paper develops optimal rates and an efficient algorithm for federated PCA and covariance estimation under privacy constraints, demonstrating robustness and scalability in distributed settings.

Contribution

It introduces the first minimax rates for federated PCA with differential privacy and proposes a scalable, near-optimal algorithm with a novel spectral decomposition approach.

Findings

Central server's rate is the harmonic mean of local rates

Estimation remains consistent even with some inconsistent local clients

Proposed algorithm achieves near-optimal rates with differential privacy

Abstract

Federated Learning (FL) has gained significant recent attention in machine learning for its enhanced privacy and data security, making it indispensable in fields such as healthcare, finance, and personalized services. This paper investigates federated PCA and estimation for spiked covariance matrices under distributed differential privacy constraints. We establish minimax rates of convergence, with a key finding that the central server's optimal rate is the harmonic mean of the local clients' minimax rates. This guarantees consistent estimation at the central server as long as at least one local client provides consistent results. Notably, consistency is maintained even if some local estimators are inconsistent, provided there are enough clients. These findings highlight the robustness and scalability of FL for reliable statistical inference under privacy constraints. To establish minimax lower bounds, we derive a matrix version of van Trees' inequality, which is of independent interest. Furthermore, we propose an efficient algorithm that preserves differential privacy while achieving near-optimal rates at the central server, up to a logarithmic factor. We address significant technical challenges in analyzing this algorithm, which involves a three-layer spectral decomposition. Numerical performance of the proposed algorithm is investigated using both simulated and real data.