Understanding of Kernels in CNN Models by Suppressing Irrelevant Visual Features in Images

TL;DR

This paper introduces a simple optimization method to interpret CNN kernels by preserving a specific kernel's activation while suppressing others, leading to clearer understanding of what visual features each kernel detects.

Contribution

The proposed method effectively isolates and visualizes features associated with individual CNN kernels, improving interpretability over existing techniques.

Findings

Better interpretation of kernel activations than existing methods.

Effective even when kernels have similar activation regions.

Visualizations highlight features specific to each kernel.

Abstract

Deep learning models have shown their superior performance in various vision tasks. However, the lack of precisely interpreting kernels in convolutional neural networks (CNNs) is becoming one main obstacle to wide applications of deep learning models in real scenarios. Although existing interpretation methods may find certain visual patterns which are associated with the activation of a specific kernel, those visual patterns may not be specific or comprehensive enough for interpretation of a specific activation of kernel of interest. In this paper, a simple yet effective optimization method is proposed to interpret the activation of any kernel of interest in CNN models. The basic idea is to simultaneously preserve the activation of the specific kernel and suppress the activation of all other kernels at the same layer. In this way, only visual information relevant to the activation of the specific kernel is remained in the input. Consistent visual information from multiple modified inputs would help users understand what kind of features are specifically associated with specific kernel. Comprehensive evaluation shows that the proposed method can help better interpret activation of specific kernels than widely used methods, even when two kernels have very similar activation regions from the same input image.