Large Scale High-Dimensional Reduced-Rank Linear Discriminant Analysis

TL;DR

This paper introduces a fast, simple iterative algorithm for high-dimensional reduced-rank linear discriminant analysis that overcomes computational challenges without extra assumptions, providing guarantees and implicit regularization.

Contribution

It proposes a novel, efficient algorithm for high-dimensional RRLDA that is free from additional assumptions and offers theoretical guarantees.

Findings

Algorithm performs well on real data

Provides implicit regularization towards least norm solution

Efficiently handles large, high-dimensional datasets

Abstract

Reduced-rank linear discriminant analysis (RRLDA) is a foundational method of dimension reduction for classification that has been useful in a wide range of applications. The goal is to identify an optimal subspace to project the observations onto that simultaneously maximizes between-group variation while minimizing within-group differences. The solution is straight forward when the number of observations is greater than the number of features but computational difficulties arise in both the high-dimensional setting, where there are more features than there are observations, and when the data are very large. Many works have proposed solutions for the high-dimensional setting and frequently involve additional assumptions or tuning parameters. We propose a fast and simple iterative algorithm for both classical and high-dimensional RRLDA on large data that is free from these additional requirements and that comes with guarantees. We also explain how RRLDA-RK provides implicit regularization towards the least norm solution without explicitly incorporating penalties. We demonstrate our algorithm on real data and highlight some results.