Linear Dimensionality Reduction

TL;DR

This paper reviews classical linear methods for dimensionality reduction, emphasizing PCA with SVD as a core technique, and discusses their relevance in modern statistical learning and large-scale data analysis.

Contribution

It provides an overview of traditional linear methods, highlighting PCA with SVD as a unifying core, and discusses recent computational improvements for large datasets.

Findings

PCA with SVD is central to many linear dimensionality reduction methods.

Efficient algorithms like Randomised SVD benefit all methods based on PCA.

Linear methods remain fundamental in statistical learning and data analysis.

Abstract

These notes are an overview of some classical linear methods in Multivariate Data Analysis. This is a good old domain, well established since the 60's, and refreshed timely as a key step in statistical learning. It can be presented as part of statistical learning, or as dimensionality reduction with a geometric flavor. Both approaches are tightly linked: it is easier to learn patterns from data in low dimensional spaces than in high-dimensional spaces. It is shown how a diversity of methods and tools boil down to a single core methods, PCA with SVD, such that the efforts to optimize codes for analyzing massive data sets like distributed memory and task-based programming or to improve the efficiency of the algorithms like Randomised SVD can focus on this shared core method, and benefit to all methods.