Do Deep Networks Transfer Invariances Across Classes?

TL;DR

This paper investigates how neural networks transfer class-agnostic invariances across classes, revealing limitations on small classes and proposing a generative approach to improve invariance transfer and classification performance in imbalanced datasets.

Contribution

The paper demonstrates the poor transfer of invariances from large to small classes and introduces a generative method to enhance invariance transfer in imbalanced image classification.

Findings

Invariance transfer is limited on small classes despite data balancing.

Neural networks' invariance to class-agnostic transformations depends heavily on class size.

Generative approaches can improve invariance transfer and classification accuracy.

Abstract

To generalize well, classifiers must learn to be invariant to nuisance transformations that do not alter an input's class. Many problems have "class-agnostic" nuisance transformations that apply similarly to all classes, such as lighting and background changes for image classification. Neural networks can learn these invariances given sufficient data, but many real-world datasets are heavily class imbalanced and contain only a few examples for most of the classes. We therefore pose the question: how well do neural networks transfer class-agnostic invariances learned from the large classes to the small ones? Through careful experimentation, we observe that invariance to class-agnostic transformations is still heavily dependent on class size, with the networks being much less invariant on smaller classes. This result holds even when using data balancing techniques, and suggests poor invariance transfer across classes. Our results provide one explanation for why classifiers generalize poorly on unbalanced and long-tailed distributions. Based on this analysis, we show how a generative approach for learning the nuisance transformations can help transfer invariances across classes and improve performance on a set of imbalanced image classification benchmarks. Source code for our experiments is available at https://github.com/AllanYangZhou/generative-invariance-transfer.