A Randomized Algorithm for Sparse PCA based on the Basic SDP Relaxation

TL;DR

This paper presents a randomized approximation algorithm for Sparse PCA based on SDP relaxation, achieving near-optimal solutions with high probability and demonstrating effectiveness on real datasets.

Contribution

Introduces a novel randomized algorithm for SPCA leveraging SDP relaxation, with proven approximation bounds and practical validation.

Findings

Approximation ratio bounded by sparsity constant with high probability.

Average approximation ratio is $O(\,log d)$ under certain assumptions.

Algorithm performs well on real-world datasets, matching theoretical predictions.

Abstract

Sparse Principal Component Analysis (SPCA) is a fundamental technique for dimensionality reduction, and is NP-hard. In this paper, we introduce a randomized approximation algorithm for SPCA, which is based on the basic SDP relaxation. Our algorithm takes an (approximate) SDP solution, constructs one deterministic sparse solution and several randomized solutions, and outputs the best among them. Our algorithm has an approximation ratio of at most the sparsity constant with high probability, if called enough times. Under a technical assumption, which is consistently satisfied in our numerical tests, the average approximation ratio is also bounded by $\mathcal{O}(\log{d})$, where $d$ is the number of features. We show that this technical assumption is satisfied if the SDP solution is low-rank, or has exponentially decaying eigenvalues. We then present two classes of instances for which this technical assumption holds. We also demonstrate that in a covariance model, which generalizes the spiked Wishart model, the deterministic solution in our algorithm achieves a near-optimal approximation ratio. We demonstrate the efficacy of our algorithm through numerical tests on real-world datasets.