Mapping Knowledge Representations to Concepts: A Review and New Perspectives

TL;DR

This paper reviews methods for linking neural network internal representations to human-understandable concepts, proposing a taxonomy based on deductive explanations and discussing the goals of model explainability.

Contribution

It introduces a taxonomy for neural network explanations using deductive reasoning and causality, and clarifies the distinction between understanding models and providing actionable explanations.

Findings

Taxonomy based on deductive explanations and causality.

Ambiguity in the goal of explainability: understanding vs. actionability.

Insights into expectations and limitations of neural network explanations.

Abstract

The success of neural networks builds to a large extent on their ability to create internal knowledge representations from real-world high-dimensional data, such as images, sound, or text. Approaches to extract and present these representations, in order to explain the neural network's decisions, is an active and multifaceted research field. To gain a deeper understanding of a central aspect of this field, we have performed a targeted review focusing on research that aims to associate internal representations with human understandable concepts. In doing this, we added a perspective on the existing research by using primarily deductive nomological explanations as a proposed taxonomy. We find this taxonomy and theories of causality, useful for understanding what can be expected, and not expected, from neural network explanations. The analysis additionally uncovers an ambiguity in the reviewed literature related to the goal of model explainability; is it understanding the ML model or, is it actionable explanations useful in the deployment domain?