A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

TL;DR

This paper introduces VRX, a framework that interprets neural network decisions using structural visual concepts, offering insights into the reasoning process and potential avenues for performance improvement.

Contribution

VRX is a novel interpretability framework that extracts class-specific visual concepts and organizes them into structural graphs to explain neural network reasoning.

Findings

VRX can answer 'why' and 'why not' questions about predictions.

VRX provides logical, concept-level explanations for model decisions.

Insights from VRX can guide neural network performance improvements.

Abstract

Despite substantial progress in applying neural networks (NN) to a wide variety of areas, they still largely suffer from a lack of transparency and interpretability. While recent developments in explainable artificial intelligence attempt to bridge this gap (e.g., by visualizing the correlation between input pixels and final outputs), these approaches are limited to explaining low-level relationships, and crucially, do not provide insights on error correction. In this work, we propose a framework (VRX) to interpret classification NNs with intuitive structural visual concepts. Given a trained classification model, the proposed VRX extracts relevant class-specific visual concepts and organizes them using structural concept graphs (SCG) based on pairwise concept relationships. By means of knowledge distillation, we show VRX can take a step towards mimicking the reasoning process of NNs and provide logical, concept-level explanations for final model decisions. With extensive experiments, we empirically show VRX can meaningfully answer "why" and "why not" questions about the prediction, providing easy-to-understand insights about the reasoning process. We also show that these insights can potentially provide guidance on improving NN's performance.