Estimating Principal Components under Adversarial Perturbations

TL;DR

This paper introduces a robust method for estimating the top principal components of high-dimensional Gaussian data under adversarial sample perturbations, generalizing existing sparse PCA results and establishing near-optimal error bounds.

Contribution

The authors propose a computationally efficient algorithm for principal component estimation under adversarial perturbations, with error bounds characterized by a new robustness parameter.

Findings

Algorithm achieves error bounds depending on the robustness parameter 

Guarantees recover existing bounds for sparse PCA in the absence of corruptions

Proves near-optimality of the error dependence on the  operator norm of the subspace

Abstract

Robustness is a key requirement for widespread deployment of machine learning algorithms, and has received much attention in both statistics and computer science. We study a natural model of robustness for high-dimensional statistical estimation problems that we call the adversarial perturbation model. An adversary can perturb every sample arbitrarily up to a specified magnitude $\delta$ measured in some $\ell_q$ norm, say $\ell_\infty$. Our model is motivated by emerging paradigms such as low precision machine learning and adversarial training. We study the classical problem of estimating the top-$r$ principal subspace of the Gaussian covariance matrix in high dimensions, under the adversarial perturbation model. We design a computationally efficient algorithm that given corrupted data, recovers an estimate of the top-$r$ principal subspace with error that depends on a robustness parameter $\kappa$ that we identify. This parameter corresponds to the $q \to 2$ operator norm of the projector onto the principal subspace, and generalizes well-studied analytic notions of sparsity. Additionally, in the absence of corruptions, our algorithmic guarantees recover existing bounds for problems such as sparse PCA and its higher rank analogs. We also prove that the above dependence on the parameter $\kappa$ is almost optimal asymptotically, not just in a minimax sense, but remarkably for every instance of the problem. This instance-optimal guarantee shows that the $q \to 2$ operator norm of the subspace essentially characterizes the estimation error under adversarial perturbations.