Approximately Equivariant Neural Processes

TL;DR

This paper introduces a flexible method to incorporate approximate equivariance into neural processes, enhancing their performance on real-world tasks where exact symmetry is not present.

Contribution

The authors develop a general, architecture-agnostic approach to achieve approximate equivariance in neural networks, applicable across different symmetry groups and models.

Findings

Approximately equivariant NPs outperform non-equivariant models.

The approach is effective on synthetic and real-world regression tasks.

It allows models to flexibly depart from strict symmetry constraints.

Abstract

Equivariant deep learning architectures exploit symmetries in learning problems to improve the sample efficiency of neural-network-based models and their ability to generalise. However, when modelling real-world data, learning problems are often not exactly equivariant, but only approximately. For example, when estimating the global temperature field from weather station observations, local topographical features like mountains break translation equivariance. In these scenarios, it is desirable to construct architectures that can flexibly depart from exact equivariance in a data-driven way. Current approaches to achieving this cannot usually be applied out-of-the-box to any architecture and symmetry group. In this paper, we develop a general approach to achieving this using existing equivariant architectures. Our approach is agnostic to both the choice of symmetry group and model architecture, making it widely applicable. We consider the use of approximately equivariant architectures in neural processes (NPs), a popular family of meta-learning models. We demonstrate the effectiveness of our approach on a number of synthetic and real-world regression experiments, showing that approximately equivariant NP models can outperform both their non-equivariant and strictly equivariant counterparts.