Neural Network Classifier as Mutual Information Evaluator

TL;DR

This paper presents a novel perspective on neural network classifiers as mutual information evaluators, introducing a new softmax form that improves accuracy on imbalanced datasets by maximizing mutual information.

Contribution

It introduces a new softmax formulation that enables neural classifiers to evaluate mutual information, especially enhancing performance on imbalanced datasets.

Findings

Maximizes mutual information between inputs and labels with cross-entropy on balanced data.

New softmax form improves classification accuracy on imbalanced datasets.

Experimental results demonstrate superior performance over standard methods.

Abstract

Cross-entropy loss with softmax output is a standard choice to train neural network classifiers. We give a new view of neural network classifiers with softmax and cross-entropy as mutual information evaluators. We show that when the dataset is balanced, training a neural network with cross-entropy maximises the mutual information between inputs and labels through a variational form of mutual information. Thereby, we develop a new form of softmax that also converts a classifier to a mutual information evaluator when the dataset is imbalanced. Experimental results show that the new form leads to better classification accuracy, in particular for imbalanced datasets.