Every Filter Extracts A Specific Texture In Convolutional Neural Networks

TL;DR

This paper introduces a feature map inversion method to analyze CNN filters, revealing that each filter extracts a specific texture, and demonstrates how image style can be composed from these textures, explaining the role of Gram matrices.

Contribution

It proposes a novel feature map inversion technique to interpret CNN filters as texture extractors and links these textures to image style representation.

Findings

Each filter extracts a unique texture.

Higher layer textures are more colorful and complex.

Style can be reconstructed from texture primitives.

Abstract

Many works have concentrated on visualizing and understanding the inner mechanism of convolutional neural networks (CNNs) by generating images that activate some specific neurons, which is called deep visualization. However, it is still unclear what the filters extract from images intuitively. In this paper, we propose a modified code inversion algorithm, called feature map inversion, to understand the function of filter of interest in CNNs. We reveal that every filter extracts a specific texture. The texture from higher layer contains more colours and more intricate structures. We also demonstrate that style of images could be a combination of these texture primitives. Two methods are proposed to reallocate energy distribution of feature maps randomly and purposefully. Then, we inverse the modified code and generate images of diverse styles. With these results, we provide an explanation about why Gram matrix of feature maps \cite{Gatys_2016_CVPR} could represent image style.