Maximum Margin Principal Components

TL;DR

This paper introduces a new dimensionality reduction method called Maximum Margin Principal Components, designed specifically for classification tasks, which often outperforms PCA by focusing on margin distribution rather than variance.

Contribution

The paper proposes a novel maximum margin-based PCA alternative tailored for classification, improving upon traditional PCA's variance-focused approach.

Findings

Our method often reduces classification error compared to PCA.

It is competitive with PLS and Lasso on various datasets.

The approach is simple and effective for classification tasks.

Abstract

Principal Component Analysis (PCA) is a very successful dimensionality reduction technique, widely used in predictive modeling. A key factor in its widespread use in this domain is the fact that the projection of a dataset onto its first $K$ principal components minimizes the sum of squared errors between the original data and the projected data over all possible rank $K$ projections. Thus, PCA provides optimal low-rank representations of data for least-squares linear regression under standard modeling assumptions. On the other hand, when the loss function for a prediction problem is not the least-squares error, PCA is typically a heuristic choice of dimensionality reduction -- in particular for classification problems under the zero-one loss. In this paper we target classification problems by proposing a straightforward alternative to PCA that aims to minimize the difference in margin distribution between the original and the projected data. Extensive experiments show that our simple approach typically outperforms PCA on any particular dataset, in terms of classification error, though this difference is not always statistically significant, and despite being a filter method is frequently competitive with Partial Least Squares (PLS) and Lasso on a wide range of datasets.