Spooky Action at a Distance: Normalization Layers Enable Side-Channel Spatial Communication

TL;DR

This paper demonstrates that normalization layers in neural networks can enable unintended spatial communication, which can impact tasks requiring strict local receptive fields, highlighting a need for cautious application.

Contribution

It reveals that normalization layers facilitate unintended spatial message passing, challenging assumptions about their neutrality in spatially sensitive tasks.

Findings

Normalization layers enable iterative message passing across spatial dimensions.

Normalization can cause information to propagate beyond local receptive fields.

Implications for applications requiring spatially limited processing.

Abstract

This work shows that normalization layers can facilitate a surprising degree of communication across the spatial dimensions of an input tensor. We study a toy localization task with a convolutional architecture and show that normalization layers enable an iterative message passing procedure, allowing information aggregation from well outside the local receptive field. Our results suggest that normalization layers should be employed with caution in applications such as diffusion-based trajectory generation, where maintaining a spatially limited receptive field is crucial.