Regularized Deep Linear Discriminant Analysis

TL;DR

This paper introduces Regularized Deep Linear Discriminant Analysis (RDLDA), a novel method that enhances deep LDA by regularizing the within-class scatter, leading to improved discriminative representations in neural networks.

Contribution

The paper proposes a regularization technique for DLDA to improve per-dimension discriminative ability and introduces Subclass RDLDA for local space enhancement.

Findings

RDLDA outperforms DLDA and traditional neural networks on multiple datasets.

Regularization improves the discriminative capacity of each dimension.

Subclass RDLDA further enhances local discriminative ability.

Abstract

As a non-linear extension of the classic Linear Discriminant Analysis(LDA), Deep Linear Discriminant Analysis(DLDA) replaces the original Categorical Cross Entropy(CCE) loss function with eigenvalue-based loss function to make a deep neural network(DNN) able to learn linearly separable hidden representations. In this paper, we first point out DLDA focuses on training the cooperative discriminative ability of all the dimensions in the latent subspace, while put less emphasis on training the separable capacity of single dimension. To improve DLDA, a regularization method on within-class scatter matrix is proposed to strengthen the discriminative ability of each dimension, and also keep them complement each other. Experiment results on STL-10, CIFAR-10 and Pediatric Pneumonic Chest X-ray Dataset showed that our proposed regularization method Regularized Deep Linear Discriminant Analysis(RDLDA) outperformed DLDA and conventional neural network with CCE as objective. To further improve the discriminative ability of RDLDA in the local space, an algorithm named Subclass RDLDA is also proposed.