Improving the Sample-Complexity of Deep Classification Networks with Invariant Integration

TL;DR

This paper introduces a novel monomial selection algorithm and alternative functions for invariant integration in deep networks, significantly reducing sample complexity and improving performance on rotation-invariant tasks.

Contribution

It proposes a pruning-based monomial selection method and replaces monomials with functions like weighted sums and self-attention, streamlining invariant integration.

Findings

Outperforms baselines in limited sample regimes on Rotated-MNIST, SVHN, and CIFAR-10.

Achieves state-of-the-art results on Rotated-MNIST and SVHN with full data.

Outperforms standard and rotation-equivariant CNNs on STL-10.

Abstract

Leveraging prior knowledge on intraclass variance due to transformations is a powerful method to improve the sample complexity of deep neural networks. This makes them applicable to practically important use-cases where training data is scarce. Rather than being learned, this knowledge can be embedded by enforcing invariance to those transformations. Invariance can be imposed using group-equivariant convolutions followed by a pooling operation. For rotation-invariance, previous work investigated replacing the spatial pooling operation with invariant integration which explicitly constructs invariant representations. Invariant integration uses monomials which are selected using an iterative approach requiring expensive pre-training. We propose a novel monomial selection algorithm based on pruning methods to allow an application to more complex problems. Additionally, we replace monomials with different functions such as weighted sums, multi-layer perceptrons and self-attention, thereby streamlining the training of invariant-integration-based architectures. We demonstrate the improved sample complexity on the Rotated-MNIST, SVHN and CIFAR-10 datasets where rotation-invariant-integration-based Wide-ResNet architectures using monomials and weighted sums outperform the respective baselines in the limited sample regime. We achieve state-of-the-art results using full data on Rotated-MNIST and SVHN where rotation is a main source of intraclass variation. On STL-10 we outperform a standard and a rotation-equivariant convolutional neural network using pooling.