Rethinking PCA Through Duality

TL;DR

This paper offers new theoretical insights and algorithms for PCA by leveraging the difference-of-convex framework, connecting classical methods to modern optimization and extending PCA to kernelized and robust variants.

Contribution

It introduces novel formulations of PCA using DC programming, reveals the connection between simultaneous iteration and DCA, and develops new algorithms including kernelized and robust PCA variants.

Findings

Kernelizability and out-of-sample extension demonstrated.

Simultaneous iteration shown as an instance of DCA.

New algorithms empirically outperform state-of-the-art methods.

Abstract

Motivated by the recently shown connection between self-attention and (kernel) principal component analysis (PCA), we revisit the fundamentals of PCA. Using the difference-of-convex (DC) framework, we present several novel formulations and provide new theoretical insights. In particular, we show the kernelizability and out-of-sample applicability for a PCA-like family of problems. Moreover, we uncover that simultaneous iteration, which is connected to the classical QR algorithm, is an instance of the difference-of-convex algorithm (DCA), offering an optimization perspective on this longstanding method. Further, we describe new algorithms for PCA and empirically compare them with state-of-the-art methods. Lastly, we introduce a kernelizable dual formulation for a robust variant of PCA that minimizes the $l_1$ deviation of the reconstruction errors.