Cyclic orthogonal convolutions for long-range integration of features

TL;DR

CycleNet introduces a novel cyclic orthogonal convolution architecture enabling efficient long-range feature integration in CNNs, improving performance on tasks requiring global context and shape recognition.

Contribution

The paper presents CycleNet, a new architecture using orthogonal convolutions in a cyclic manner to enhance long-range feature integration with minimal layers.

Findings

Competitive image classification results on CIFAR-10 and ImageNet.

Superior performance on the Pathfinder challenge.

Receptive fields reach maximum size after one cycle, unlike traditional CNNs.

Abstract

In Convolutional Neural Networks (CNNs) information flows across a small neighbourhood of each pixel of an image, preventing long-range integration of features before reaching deep layers in the network. We propose a novel architecture that allows flexible information flow between features $z$ and locations $(x,y)$ across the entire image with a small number of layers. This architecture uses a cycle of three orthogonal convolutions, not only in $(x,y)$ coordinates, but also in $(x,z)$ and $(y,z)$ coordinates. We stack a sequence of such cycles to obtain our deep network, named CycleNet. As this only requires a permutation of the axes of a standard convolution, its performance can be directly compared to a CNN. Our model obtains competitive results at image classification on CIFAR-10 and ImageNet datasets, when compared to CNNs of similar size. We hypothesise that long-range integration favours recognition of objects by shape rather than texture, and we show that CycleNet transfers better than CNNs to stylised images. On the Pathfinder challenge, where integration of distant features is crucial, CycleNet outperforms CNNs by a large margin. We also show that even when employing a small convolutional kernel, the size of receptive fields of CycleNet reaches its maximum after one cycle, while conventional CNNs require a large number of layers.