Simplex Deep Linear Discriminant Analysis

TL;DR

This paper revisits Deep Linear Discriminant Analysis, identifying issues with unconstrained training and proposing a geometric constraint-based formulation that stabilizes training and produces interpretable, well-separated class clusters.

Contribution

It introduces a constrained Deep LDA model with geometric restrictions that enable stable maximum likelihood training and interpretable latent space representations.

Findings

Achieves competitive accuracy on image and text datasets.

Produces clearly visible, well-separated class clusters in the latent space.

Addresses instability issues in unconstrained Deep LDA training.

Abstract

We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of how to train the resulting deep classifier by maximum likelihood estimation (MLE). We first show that end-to-end MLE training of an unconstrained Deep LDA model ignores discrimination: when both the LDA parameters and the encoder parameters are learned jointly, the likelihood admits a degenerate solution in which some of the class clusters may heavily overlap or even collapse, and classification performance deteriorates. Batchwise moment re-estimation of the LDA parameters does not remove this failure mode. We then propose a constrained Deep LDA formulation that fixes the class means to the vertices of a regular simplex in the latent space and restricts the shared covariance to be spherical, leaving only the priors and a single variance parameter to be learned along with the encoder. Under these geometric constraints, MLE becomes stable and yields well-separated class clusters in the latent space. On images (Fashion-MNIST, CIFAR-10, CIFAR-100) and texts (AG News, CLINC150), the resulting Deep LDA models achieve accuracy competitive with softmax baselines while offering a simple, interpretable latent geometry that is clearly visible in two-dimensional projections.