Autoequivariant Network Search via Group Decomposition

TL;DR

This paper introduces autoequivariant networks (AENs) that automatically find optimal group equivariances for neural networks, balancing performance and size, using a novel group-theoretic approach and deep Q-learning, tested on new benchmark datasets.

Contribution

It presents a new theoretical result linking large and small group equivariances, and develops a fast construction and search algorithm for AENs, advancing automated equivariance design.

Findings

AENs effectively balance equivariance and network size.

Group equivariance is the most influential factor in high-performing GCNNs.

New benchmark datasets G-MNIST and G-Fashion-MNIST demonstrate the approach.

Abstract

Recent works show that group equivariance as an inductive bias improves neural network performance for both classification and generation. However, designing group-equivariant neural networks is challenging when the group of interest is large and is unknown. Moreover, inducing equivariance can significantly reduce the number of independent parameters in a network with fixed feature size, affecting its overall performance. We address these problems by proving a new group-theoretic result in the context of equivariant neural networks that shows that a network is equivariant to a large group if and only if it is equivariant to smaller groups from which it is constructed. Using this result, we design a novel fast group equivariant construction algorithm, and a deep Q-learning-based search algorithm in a reduced search space, yielding what we call autoequivariant networks (AENs). AENs find the right balance between equivariance and network size when tested on new benchmark datasets, G-MNIST and G-Fashion-MNIST, obtained via group transformations on MNIST and Fashion-MNIST respectively that we release. Extending these results to group convolutional neural networks, where we optimize between equivariances, augmentations, and network sizes, we find group equivariance to be the most dominating factor in all high-performing GCNNs on several datasets like CIFAR10, SVHN, RotMNIST, ASL, EMNIST, and KMNIST.