What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective

TL;DR

This paper investigates the first layer representations of CNNs, revealing their consistent energy distribution across initializations and architectures, and explains this phenomenon through a linear systems perspective that relates to whitening transformations.

Contribution

It provides an analytical formula for the energy profile of linear CNNs and demonstrates its applicability to nonlinear CNNs like ResNet and VGG, explaining their whitening behavior.

Findings

First layer energy distribution is highly consistent across initializations and architectures.

The energy profile is mainly determined by second order statistics of image patches.

Linear CNNs' energy profile approaches a whitening transformation with training iterations.

Abstract

It has previously been reported that the representation that is learned in the first layer of deep Convolutional Neural Networks (CNNs) is highly consistent across initializations and architectures. In this work, we quantify this consistency by considering the first layer as a filter bank and measuring its energy distribution. We find that the energy distribution is very different from that of the initial weights and is remarkably consistent across random initializations, datasets, architectures and even when the CNNs are trained with random labels. In order to explain this consistency, we derive an analytical formula for the energy profile of linear CNNs and show that this profile is mostly dictated by the second order statistics of image patches in the training set and it will approach a whitening transformation when the number of iterations goes to infinity. Finally, we show that this formula for linear CNNs also gives an excellent fit for the energy profiles learned by commonly used nonlinear CNNs such as ResNet and VGG, and that the first layer of these CNNs indeed perform approximate whitening of their inputs.