Discriminative Dimension Reduction based on Mutual Information

TL;DR

This paper introduces an information theory-based method for selecting discriminative subspaces using mutual information, significantly improving pattern classification performance across various tasks.

Contribution

It develops a novel mutual information criterion for subspace selection tailored for classification, outperforming traditional methods.

Findings

Subspaces selected by maximum mutual information improve classification accuracy.

The method is effective across diverse computer vision and pattern recognition tasks.

Performance enhancement is consistent regardless of the classifier used.

Abstract

The "curse of dimensionality" is a well-known problem in pattern recognition. A widely used approach to tackling the problem is a group of subspace methods, where the original features are projected onto a new space. The lower dimensional subspace is then used to approximate the original features for classification. However, most subspace methods were not originally developed for classification. We believe that direct adoption of these subspace methods for pattern classification should not be considered best practice. In this paper, we present a new information theory based algorithm for selecting subspaces, which can always result in superior performance over conventional methods. This paper makes the following main contributions: i) it improves a common practice widely used by practitioners in the field of pattern recognition, ii) it develops an information theory based technique for systematically selecting the subspaces that are discriminative and therefore are suitable for pattern recognition/classification purposes, iii) it presents extensive experimental results on a variety of computer vision and pattern recognition tasks to illustrate that the subspaces selected based on maximum mutual information criterion will always enhance performance regardless of the classification techniques used.