Conditional Mutual Information Constrained Deep Learning for Classification

TL;DR

This paper introduces a new information-theoretic framework using conditional mutual information to improve deep neural network training, resulting in better accuracy and robustness against adversarial attacks.

Contribution

It proposes CMI and NCMI as measures for DNN performance, and develops a constrained optimization method (CMIC-DL) with an alternating algorithm for enhanced training.

Findings

CMIC-DL outperforms standard training in accuracy.

Models trained with CMIC-DL show increased robustness.

NCMI correlates inversely with validation accuracy.

Abstract

The concepts of conditional mutual information (CMI) and normalized conditional mutual information (NCMI) are introduced to measure the concentration and separation performance of a classification deep neural network (DNN) in the output probability distribution space of the DNN, where CMI and the ratio between CMI and NCMI represent the intra-class concentration and inter-class separation of the DNN, respectively. By using NCMI to evaluate popular DNNs pretrained over ImageNet in the literature, it is shown that their validation accuracies over ImageNet validation data set are more or less inversely proportional to their NCMI values. Based on this observation, the standard deep learning (DL) framework is further modified to minimize the standard cross entropy function subject to an NCMI constraint, yielding CMI constrained deep learning (CMIC-DL). A novel alternating learning algorithm is proposed to solve such a constrained optimization problem. Extensive experiment results show that DNNs trained within CMIC-DL outperform the state-of-the-art models trained within the standard DL and other loss functions in the literature in terms of both accuracy and robustness against adversarial attacks. In addition, visualizing the evolution of learning process through the lens of CMI and NCMI is also advocated.