HyperInvariances: Amortizing Invariance Learning

TL;DR

HyperInvariances introduces a framework that amortizes invariance learning by pre-training a manifold of feature extractors, enabling rapid adaptation to various tasks and improving generalization with theoretical guarantees.

Contribution

The paper proposes a novel amortized invariance learning method using hyper-networks, reducing data and computational costs for invariance adaptation in neural networks.

Findings

Empirically identifies appropriate invariances across tasks.

Achieves comparable or better test performance than traditional methods.

Provides theoretical generalization bounds for the approach.

Abstract

Providing invariances in a given learning task conveys a key inductive bias that can lead to sample-efficient learning and good generalisation, if correctly specified. However, the ideal invariances for many problems of interest are often not known, which has led both to a body of engineering lore as well as attempts to provide frameworks for invariance learning. However, invariance learning is expensive and data intensive for popular neural architectures. We introduce the notion of amortizing invariance learning. In an up-front learning phase, we learn a low-dimensional manifold of feature extractors spanning invariance to different transformations using a hyper-network. Then, for any problem of interest, both model and invariance learning are rapid and efficient by fitting a low-dimensional invariance descriptor an output head. Empirically, this framework can identify appropriate invariances in different downstream tasks and lead to comparable or better test performance than conventional approaches. Our HyperInvariance framework is also theoretically appealing as it enables generalisation-bounds that provide an interesting new operating point in the trade-off between model fit and complexity.