Divergence Maximizing Linear Projection for Supervised Dimension Reduction

TL;DR

This paper introduces two supervised linear projection methods for dimension reduction based on maximizing the Kullback-Leibler divergence, leveraging Gaussian assumptions and unifying existing approaches while establishing optimality conditions.

Contribution

It develops novel linear projection techniques for supervised dimension reduction that maximize KL divergence, generalizing and unifying existing methods under Gaussian models.

Findings

Proposed methods outperform existing linear projection approaches.

Established the optimality of multi-class linear discriminant analysis under certain conditions.

Validated the methods through experiments demonstrating superior performance.

Abstract

This paper proposes two linear projection methods for supervised dimension reduction using only the first and second-order statistics. The methods, each catering to a different parameter regime, are derived under the general Gaussian model by maximizing the Kullback-Leibler divergence between the two classes in the projected sample for a binary classification problem. They subsume existing linear projection approaches developed under simplifying assumptions of Gaussian distributions, such as these distributions might share an equal mean or covariance matrix. As a by-product, we establish that the multi-class linear discriminant analysis, a celebrated method for classification and supervised dimension reduction, is provably optimal for maximizing pairwise Kullback-Leibler divergence when the Gaussian populations share an identical covariance matrix. For the case when the Gaussian distributions share an equal mean, we establish conditions under which the optimal subspace remains invariant regardless of how the Kullback-Leibler divergence is defined, despite the asymmetry of the divergence measure itself. Such conditions encompass the classical case of signal plus noise, where both the signal and noise have zero mean and arbitrary covariance matrices. Experiments are conducted to validate the proposed solutions, demonstrate their superior performance over existing alternatives, and illustrate the procedure for selecting the appropriate linear projection solution.