Explaining Classifiers by Constructing Familiar Concepts

TL;DR

This paper introduces ClaDec, a neural network architecture that helps interpret deep learning models by reconstructing inputs in familiar concepts, making it easier for humans to understand what features are used for classification.

Contribution

ClaDec is a novel architecture that transforms neuron representations into human-understandable concepts, improving interpretability of deep neural networks.

Findings

Reconstructed images with ClaDec capture more relevant classification concepts than traditional auto-encoders.

Even non-experts can identify key concepts in images using ClaDec visualizations.

ClaDec highlights more relevant input areas compared to saliency-based methods.

Abstract

Interpreting a large number of neurons in deep learning is difficult. Our proposed `CLAssifier-DECoder' architecture (ClaDec) facilitates the understanding of the output of an arbitrary layer of neurons or subsets thereof. It uses a decoder that transforms the incomprehensible representation of the given neurons to a representation that is more similar to the domain a human is familiar with. In an image recognition problem, one can recognize what information (or concepts) a layer maintains by contrasting reconstructed images of ClaDec with those of a conventional auto-encoder(AE) serving as reference. An extension of ClaDec allows trading comprehensibility and fidelity. We evaluate our approach for image classification using convolutional neural networks. We show that reconstructed visualizations using encodings from a classifier capture more relevant classification information than conventional AEs. This holds although AEs contain more information on the original input. Our user study highlights that even non-experts can identify a diverse set of concepts contained in images that are relevant (or irrelevant) for the classifier. We also compare against saliency based methods that focus on pixel relevance rather than concepts. We show that ClaDec tends to highlight more relevant input areas to classification though outcomes depend on classifier architecture. Code is at \url{https://github.com/JohnTailor/ClaDec}