Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies

TL;DR

This paper introduces a novel regularization technique based on functional entropy to balance modality contributions in multi-modal classifiers, improving fairness and performance on datasets like VQA-CPv2 and SocialIQ.

Contribution

It proposes a new regularization method using functional entropy and the log-Sobolev inequality to enhance multi-modal classifier fairness and effectiveness.

Findings

Achieved state-of-the-art results on VQA-CPv2 and SocialIQ datasets.

Effectively balanced modality contributions in multi-modal classification.

Demonstrated improved performance on Colored MNIST.

Abstract

Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to balance the contribution of each modality to the classification result. However, regularization with the functional entropy is challenging. To address this, we develop a method based on the log-Sobolev inequality, which bounds the functional entropy with the functional-Fisher-information. Intuitively, this maximizes the amount of information that the modalities contribute. On the two challenging multi-modal datasets VQA-CPv2 and SocialIQ, we obtain state-of-the-art results while more uniformly exploiting the modalities. In addition, we demonstrate the efficacy of our method on Colored MNIST.