Robust covariance estimation for distributed principal component analysis

TL;DR

This paper improves distributed PCA algorithms by incorporating robust covariance estimators, enabling effective analysis of heavy-tailed data with theoretical guarantees and strong empirical performance.

Contribution

It introduces a robust distributed PCA method using advanced covariance estimators to handle heavy-tailed data, extending guarantees beyond sub-Gaussian assumptions.

Findings

The algorithm achieves near sub-Gaussian error rates for heavy-tailed distributions.

Numerical experiments confirm robustness to outliers and heavy tails.

Theoretical analysis supports effectiveness under symmetric and asymmetric heavy-tailed conditions.

Abstract

Fan et al. [$\mathit{Annals}$ $\mathit{of}$ $\mathit{Statistics}$ $\textbf{47}$(6) (2019) 3009-3031] constructed a distributed principal component analysis (PCA) algorithm to reduce the communication cost between multiple servers significantly. However, their algorithm's guarantee is only for sub-Gaussian data. Spurred by this deficiency, this paper enhances the effectiveness of their distributed PCA algorithm by utilizing robust covariance matrix estimators of Minsker [$\mathit{Annals}$ $\mathit{of}$ $\mathit{Statistics}$ $\textbf{46}$(6A) (2018) 2871-2903] and Ke et al. [$\mathit{Statistical}$ $\mathit{Science}$ $\textbf{34}$(3) (2019) 454-471] to tame heavy-tailed data. The theoretical results demonstrate that when the sampling distribution is symmetric innovation with the bounded fourth moment or asymmetric with the finite $6$-th moment, the statistical error rate of the final estimator produced by the robust algorithm is similar to that of sub-Gaussian tails. Extensive numerical trials support the theoretical analysis and indicate that our algorithm is robust to heavy-tailed data and outliers.