Theoretical Guarantees for Sparse Principal Component Analysis based on the Elastic Net

TL;DR

This paper provides the first theoretical guarantees for the convergence and consistency of the popular SPCA algorithm based on the Elastic Net, under a sparse spiked covariance model, with matching error bounds and competitive numerical results.

Contribution

It establishes convergence and recovery guarantees for Zou et al.'s SPCA algorithm and its variant, filling a significant gap in the theoretical understanding of this widely used method.

Findings

Both algorithms converge to a stationary point.

They can recover the principal subspace consistently.

Estimation error bounds match minimax rates up to logarithmic factors.

Abstract

Sparse principal component analysis (SPCA) is widely used for dimensionality reduction and feature extraction in high-dimensional data analysis. Despite many methodological and theoretical developments in the past two decades, the theoretical guarantees of the popular SPCA algorithm proposed by Zou, Hastie & Tibshirani (2006) are still unknown. This paper aims to address this critical gap. We first revisit the SPCA algorithm of Zou et al. (2006) and present our implementation. We also study a computationally more efficient variant of the SPCA algorithm in Zou et al. (2006) that can be considered as the limiting case of SPCA. We provide the guarantees of convergence to a stationary point for both algorithms and prove that, under a sparse spiked covariance model, both algorithms can recover the principal subspace consistently under mild regularity conditions. We show that their estimation error bounds match the best available bounds of existing works or the minimax rates up to some logarithmic factors. Moreover, we demonstrate the competitive numerical performance of both algorithms in numerical studies.