What do CNN neurons learn: Visualization & Clustering

TL;DR

This paper explores methods to interpret CNN neurons by visualizing and clustering their activations, revealing what features neurons learn and how they contribute to predictions.

Contribution

It introduces visualization and clustering techniques to interpret CNN neurons, providing insights into their learned features and decision-making process.

Findings

Visualization reveals neuron focus areas.

Clustering groups neurons by learned features.

Methods effectively explain CNN neuron functions.

Abstract

In recent years convolutional neural networks (CNN) have shown striking progress in various tasks. However, despite the high performance, the training and prediction process remains to be a black box, leaving it a mystery to extract what neurons learn in CNN. In this paper, we address the problem of interpreting a CNN from the aspects of the input image's focus and preference, and the neurons' domination, activation and contribution to a concrete final prediction. Specifically, we use two techniques - visualization and clustering - to tackle the problems above. Visualization means the method of gradient descent on image pixel, and in clustering section two algorithms are proposed to cluster respectively over image categories and network neurons. Experiments and quantitative analyses have demonstrated the effectiveness of the two methods in explaining the question: what do neurons learn.