How deep convolutional neural networks lose spatial information with training

TL;DR

This paper investigates how deep convolutional neural networks lose spatial information during training, revealing the roles of pooling and ReLU units in sensitivity to diffeomorphisms and noise, with empirical and analytical insights.

Contribution

It provides empirical evidence and an analytical model explaining how spatial pooling and strides influence sensitivity loss and noise amplification in deep CNNs.

Findings

Spatial and channel pooling achieve stability to diffeomorphisms.

Sensitivity to noise increases with depth due to pooling and ReLU effects.

Strides in architecture affect the scaling of sensitivity to transformations.

Abstract

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the net. This loss of sensitivity correlates with performance, and surprisingly correlates with a gain of sensitivity to white noise acquired during training. These facts are unexplained, and as we demonstrate still hold when white noise is added to the images of the training set. Here, we (i) show empirically for various architectures that stability to image diffeomorphisms is achieved by both spatial and channel pooling, (ii) introduce a model scale-detection task which reproduces our empirical observations on spatial pooling and (iii) compute analitically how the sensitivity to diffeomorphisms and noise scales with depth due to spatial pooling. The scalings are found to depend on the presence of strides in the net architecture. We find that the increased sensitivity to noise is due to the perturbing noise piling up during pooling, after being rectified by ReLU units.