Why Convolutional Networks Learn Oriented Bandpass Filters: Theory and Empirical Support

TL;DR

This paper provides a theoretical explanation and empirical evidence that convolutional networks learn oriented bandpass filters at all layers, driven by the local operation of convolutions and the eigenfunctions of complex exponentials.

Contribution

It introduces a novel theory linking convolutional filters to eigenfunctions of complex exponentials and demonstrates that such filters are learned throughout all layers of CNNs.

Findings

Convolutional eigenfunctions are globally defined complex exponentials.

Local windowing of eigenfunctions leads to oriented bandpass filters.

Empirical evidence shows all CNN layers learn oriented bandpass filters.

Abstract

It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass filters. A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. We offer an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures. In particular, complex exponentials are the eigenfunctions of convolution. These eigenfunctions are defined globally; however, convolutional architectures operate locally. To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass filters as the natural operators to be learned with convolutional architectures. From a representational point of view, these filters allow for a local systematic way to characterize and operate on an image or other signal. We offer empirical support for the hypothesis that convolutional networks learn such filters at all of their convolutional layers. While previous research has shown evidence of filters having oriented bandpass characteristics at early layers, ours appears to be the first study to document the predominance of such filter characteristics at all layers. Previous studies have missed this observation because they have concentrated on the cumulative compositional effects of filtering across layers, while we examine the filter characteristics that are present at each layer.