Learning to Transform for Generalizable Instance-wise Invariance

TL;DR

This paper introduces a method that learns to predict and adapt invariance transformations for each image using normalizing flows, improving robustness and accuracy in image classification tasks.

Contribution

It proposes a novel approach to learn instance-specific invariance distributions with normalizing flows, enabling better generalization and adaptation to out-of-distribution poses.

Findings

Improves accuracy on CIFAR 10, CIFAR10-LT, TinyImageNet

Enhances robustness to spatial transformations

Learns a wider range of transformations than previous methods

Abstract

Computer vision research has long aimed to build systems that are robust to spatial transformations found in natural data. Traditionally, this is done using data augmentation or hard-coding invariances into the architecture. However, too much or too little invariance can hurt, and the correct amount is unknown a priori and dependent on the instance. Ideally, the appropriate invariance would be learned from data and inferred at test-time. We treat invariance as a prediction problem. Given any image, we use a normalizing flow to predict a distribution over transformations and average the predictions over them. Since this distribution only depends on the instance, we can align instances before classifying them and generalize invariance across classes. The same distribution can also be used to adapt to out-of-distribution poses. This normalizing flow is trained end-to-end and can learn a much larger range of transformations than Augerino and InstaAug. When used as data augmentation, our method shows accuracy and robustness gains on CIFAR 10, CIFAR10-LT, and TinyImageNet.