Pitfalls of Conditional Batch Normalization for Contextual Multi-Modal Learning

TL;DR

This paper critically examines Conditional Batch Normalization (CBN) in multi-modal learning, revealing that CBN can hinder visual feature learning and may lead to shortcut learning, thus questioning its effectiveness for generalization.

Contribution

The study provides a comprehensive evaluation of CBN's limitations across multiple datasets, highlighting its potential to impair visual feature learning and suggesting alternative approaches for better generalization.

Findings

CBN deteriorates visual feature learning in multi-modal tasks.

CBN networks show limited visual feature extraction on bird and histology datasets.

CBN may promote shortcut learning between auxiliary data and labels.

Abstract

Humans have perfected the art of learning from multiple modalities through sensory organs. Despite their impressive predictive performance on a single modality, neural networks cannot reach human level accuracy with respect to multiple modalities. This is a particularly challenging task due to variations in the structure of respective modalities. Conditional Batch Normalization (CBN) is a popular method that was proposed to learn contextual features to aid deep learning tasks. This technique uses auxiliary data to improve representational power by learning affine transformations for convolutional neural networks. Despite the boost in performance observed by using CBN layers, our work reveals that the visual features learned by introducing auxiliary data via CBN deteriorates. We perform comprehensive experiments to evaluate the brittleness of CBN networks to various datasets, suggesting that learning from visual features alone could often be superior for generalization. We evaluate CBN models on natural images for bird classification and histology images for cancer type classification. We observe that the CBN network learns close to no visual features on the bird classification dataset and partial visual features on the histology dataset. Our extensive experiments reveal that CBN may encourage shortcut learning between the auxiliary data and labels.