Understanding learned CNN features through Filter Decoding with Substitution

TL;DR

This paper introduces a novel filter substitution method to better understand and visualize CNN internal features, enabling direct visualization of individual neurons as filters in the image space.

Contribution

It proposes a new assumption based on filter substitution for reversing CNN layer encoding, improving interpretability over traditional deconvolutional approaches.

Findings

Enables direct visualization of CNN neurons as filters in image space

Provides a more accurate approximation of inverse convolution

Enhances understanding of CNN internal representations

Abstract

In parallel with the success of CNNs to solve vision problems, there is a growing interest in developing methodologies to understand and visualize the internal representations of these networks. How the responses of a trained CNN encode the visual information is a fundamental question both for computer and human vision research. Image representations provided by the first convolutional layer as well as the resolution change provided by the max-polling operation are easy to understand, however, as soon as a second and further convolutional layers are added in the representation, any intuition is lost. A usual way to deal with this problem has been to define deconvolutional networks that somehow allow to explore the internal representations of the most important activations towards the image space, where deconvolution is assumed as a convolution with the transposed filter. However, this assumption is not the best approximation of an inverse convolution. In this paper we propose a new assumption based on filter substitution to reverse the encoding of a convolutional layer. This provides us with a new tool to directly visualize any CNN single neuron as a filter in the first layer, this is in terms of the image space.