Understanding Intra-Class Knowledge Inside CNN

TL;DR

This paper visualizes intra-class knowledge within CNNs to understand how object classes are represented internally, introducing a non-parametric patch prior to generate interpretable images and exploring applications like style-based retrieval.

Contribution

It presents a novel method to visualize intra-class knowledge in CNNs using a non-parametric patch prior, revealing hierarchical and ensemble representations of object styles.

Findings

Different object styles are organized by CNN in terms of location and content.

Intra-class knowledge can be used for style-based image retrieval.

The method produces interpretable images of intra-class variations.

Abstract

Convolutional Neural Network (CNN) has been successful in image recognition tasks, and recent works shed lights on how CNN separates different classes with the learned inter-class knowledge through visualization. In this work, we instead visualize the intra-class knowledge inside CNN to better understand how an object class is represented in the fully-connected layers. To invert the intra-class knowledge into more interpretable images, we propose a non-parametric patch prior upon previous CNN visualization models. With it, we show how different "styles" of templates for an object class are organized by CNN in terms of location and content, and represented in a hierarchical and ensemble way. Moreover, such intra-class knowledge can be used in many interesting applications, e.g. style-based image retrieval and style-based object completion.