Deep Linear Discriminant Analysis

TL;DR

DeepLDA integrates linear discriminant analysis into deep neural networks to learn linearly separable features end-to-end, improving class separation and performance on benchmark datasets.

Contribution

This paper introduces DeepLDA, a novel method combining LDA with deep learning, enabling end-to-end training for better class separation.

Findings

DeepLDA achieves competitive results on MNIST and CIFAR-10.

Outperforms standard networks on STL-10.

Allows stochastic gradient training with an LDA-based objective.

Abstract

We introduce Deep Linear Discriminant Analysis (DeepLDA) which learns linearly separable latent representations in an end-to-end fashion. Classic LDA extracts features which preserve class separability and is used for dimensionality reduction for many classification problems. The central idea of this paper is to put LDA on top of a deep neural network. This can be seen as a non-linear extension of classic LDA. Instead of maximizing the likelihood of target labels for individual samples, we propose an objective function that pushes the network to produce feature distributions which: (a) have low variance within the same class and (b) high variance between different classes. Our objective is derived from the general LDA eigenvalue problem and still allows to train with stochastic gradient descent and back-propagation. For evaluation we test our approach on three different benchmark datasets (MNIST, CIFAR-10 and STL-10). DeepLDA produces competitive results on MNIST and CIFAR-10 and outperforms a network trained with categorical cross entropy (same architecture) on a supervised setting of STL-10.