Linear Discriminant Analysis with the Randomized Kaczmarz Method

TL;DR

This paper introduces a randomized Kaczmarz method for linear discriminant analysis that efficiently handles large datasets, achieving comparable accuracy to traditional LDA through an iterative stochastic approach.

Contribution

The paper develops a novel randomized Kaczmarz-based algorithm for LDA, providing theoretical analysis and demonstrating its effectiveness on large-scale data.

Findings

rkLDA achieves comparable accuracy to full data LDA.

The method's performance depends on data condition number and model fit.

Experiments show rkLDA as a viable large-scale LDA alternative.

Abstract

We present a randomized Kaczmarz method for linear discriminant analysis (rkLDA), an iterative randomized approach to binary-class Gaussian model linear discriminant analysis (LDA) for very large data. We harness a least squares formulation and mobilize the stochastic gradient descent framework to obtain a randomized classifier with performance that can achieve comparable accuracy to that of full data LDA. We present analysis for the expected change in the LDA discriminant function if one employs the randomized Kaczmarz solution in lieu of the full data least squares solution that accounts for both the Gaussian modeling assumptions on the data and algorithmic randomness. Our analysis shows how the expected change depends on quantities inherent in the data such as the scaled condition number and Frobenius norm of the input data, how well the linear model fits the data, and choices from the randomized algorithm. Our experiments demonstrate that rkLDA can offer a viable alternative to full data LDA on a range of step-sizes and numbers of iterations.