Unsupervised and Supervised Principal Component Analysis: Tutorial

TL;DR

This tutorial comprehensively explains various PCA methods, including supervised and kernel variants, detailing their mathematical foundations, properties, and practical applications with simulations on face datasets.

Contribution

It provides an in-depth, unified tutorial on PCA, supervised PCA, and kernel PCA methods, including their derivations, properties, and implementation details.

Findings

Kernel PCA effectively captures nonlinear structures.

Supervised PCA improves feature relevance for specific tasks.

Simulations validate theoretical concepts on face datasets.

Abstract

This is a detailed tutorial paper which explains the Principal Component Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start with projection, PCA with eigen-decomposition, PCA with one and multiple projection directions, properties of the projection matrix, reconstruction error minimization, and we connect to autoencoder. Then, PCA with singular value decomposition, dual PCA, and kernel PCA are covered. SPCA using both scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA using both direct and dual approaches are then introduced. We cover all cases of projection and reconstruction of training and out-of-sample data. Finally, some simulations are provided on Frey and AT&T face datasets for verifying the theory in practice.