Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision

TL;DR

This paper introduces an unsupervised approach to uncover distributed concept representations in deep neural networks by selecting principal neurons, enabling interpretation of model behavior without human-labeled concepts.

Contribution

The novel method identifies principal neurons to form interpretable regions, revealing layered concept representations and aiding in understanding model decisions without supervision.

Findings

Instances with similar neuron activation share coherent concepts

The method detects unlabeled subclasses and misclassification causes

Distributed representations vary across network layers

Abstract

Understanding intermediate representations of the concepts learned by deep learning classifiers is indispensable for interpreting general model behaviors. Existing approaches to reveal learned concepts often rely on human supervision, such as pre-defined concept sets or segmentation processes. In this paper, we propose a novel unsupervised method for discovering distributed representations of concepts by selecting a principal subset of neurons. Our empirical findings demonstrate that instances with similar neuron activation states tend to share coherent concepts. Based on the observations, the proposed method selects principal neurons that construct an interpretable region, namely a Relaxed Decision Region (RDR), encompassing instances with coherent concepts in the feature space. It can be utilized to identify unlabeled subclasses within data and to detect the causes of misclassifications. Furthermore, the applicability of our method across various layers discloses distinct distributed representations over the layers, which provides deeper insights into the internal mechanisms of the deep learning model.