Maximum mutual information regularized classification

TL;DR

This paper introduces a new classification method that enhances accuracy by maximizing mutual information between classifier responses and true labels, leading to better uncertainty reduction and improved performance.

Contribution

It proposes a novel regularization technique that maximizes mutual information during classifier training, which is a new approach compared to traditional methods.

Findings

Significant performance improvements on real-world datasets.

Effective mutual information estimation via entropy modeling.

Enhanced classifier robustness and accuracy.

Abstract

In this paper, a novel pattern classification approach is proposed by regularizing the classifier learning to maximize mutual information between the classification response and the true class label. We argue that, with the learned classifier, the uncertainty of the true class label of a data sample should be reduced by knowing its classification response as much as possible. The reduced uncertainty is measured by the mutual information between the classification response and the true class label. To this end, when learning a linear classifier, we propose to maximize the mutual information between classification responses and true class labels of training samples, besides minimizing the classification error and reduc- ing the classifier complexity. An objective function is constructed by modeling mutual information with entropy estimation, and it is optimized by a gradi- ent descend method in an iterative algorithm. Experiments on two real world pattern classification problems show the significant improvements achieved by maximum mutual information regularization.