Learning Canonical Transformations

TL;DR

This paper investigates how neural networks can learn canonical geometric transformations like translation and rotation in pixel space, emphasizing the importance of training diversity and iterative training for out-of-domain generalization.

Contribution

It demonstrates that high training diversity enables translation extrapolation and iterative training improves rotation generalization in neural networks.

Findings

High training diversity suffices for translation extrapolation.

Iterative training enhances rotation generalization.

Neural networks can learn canonical transformations in pixel space.

Abstract

Humans understand a set of canonical geometric transformations (such as translation and rotation) that support generalization by being untethered to any specific object. We explore inductive biases that help a neural network model learn these transformations in pixel space in a way that can generalize out-of-domain. Specifically, we find that high training set diversity is sufficient for the extrapolation of translation to unseen shapes and scales, and that an iterative training scheme achieves significant extrapolation of rotation in time.