4-Connected Shift Residual Networks

TL;DR

This paper explores the use of 4-connected shift operations in residual networks, demonstrating that they can match or surpass traditional convolutions in accuracy while significantly reducing computational costs.

Contribution

It introduces a 4-connected shift residual network that achieves high accuracy with fewer parameters and FLOPs, outperforming 8-connected shifts and traditional ResNet structures.

Findings

4-connected shifts match 8-connected shifts in accuracy on ImageNet.

4-connected shifts outperform 8-connected shifts when applied to all point-wise convolutions.

The proposed shift-based network achieves higher accuracy with over 40% fewer parameters and FLOPs.

Abstract

The shift operation was recently introduced as an alternative to spatial convolutions. The operation moves subsets of activations horizontally and/or vertically. Spatial convolutions are then replaced with shift operations followed by point-wise convolutions, significantly reducing computational costs. In this work, we investigate how shifts should best be applied to high accuracy CNNs. We apply shifts of two different neighbourhood groups to ResNet on ImageNet: the originally introduced 8-connected (8C) neighbourhood shift and the less well studied 4-connected (4C) neighbourhood shift. We find that when replacing ResNet's spatial convolutions with shifts, both shift neighbourhoods give equal ImageNet accuracy, showing the sufficiency of small neighbourhoods for large images. Interestingly, when incorporating shifts to all point-wise convolutions in residual networks, 4-connected shifts outperform 8-connected shifts. Such a 4-connected shift setup gives the same accuracy as full residual networks while reducing the number of parameters and FLOPs by over 40%. We then highlight that without spatial convolutions, ResNet's downsampling/upsampling bottleneck channel structure is no longer needed. We show a new, 4C shift-based residual network, much shorter than the original ResNet yet with a higher accuracy for the same computational cost. This network is the highest accuracy shift-based network yet shown, demonstrating the potential of shifting in deep neural networks.