Translation-Equivariance of Normalization Layers and Aliasing in Convolutional Neural Networks

TL;DR

This paper develops a theoretical framework to analyze the translation-equivariance properties of normalization layers in CNNs, providing conditions for equivariance and validating them empirically with ResNet-18 on ImageNet.

Contribution

It introduces a novel theoretical approach to understand normalization layers' equivariance and establishes necessary and sufficient conditions for their translation-equivariance.

Findings

Normalization layers can be designed to be equivariant under certain conditions.

Empirical results on ResNet-18 support the theoretical predictions.

The framework guides the design of more physically accurate CNN architectures.

Abstract

The design of convolutional neural architectures that are exactly equivariant to continuous translations is an active field of research. It promises to benefit scientific computing, notably by making existing imaging systems more physically accurate. Most efforts focus on the design of downsampling/pooling layers, upsampling layers and activation functions, but little attention is dedicated to normalization layers. In this work, we present a novel theoretical framework for understanding the equivariance of normalization layers to discrete shifts and continuous translations. We also determine necessary and sufficient conditions for normalization layers to be equivariant in terms of the dimensions they operate on. Using real feature maps from ResNet-18 and ImageNet, we test those theoretical results empirically and find that they are consistent with our predictions.