Coloring black boxes: visualization of neural network decisions

TL;DR

This paper introduces a visualization method for neural networks that maps high-dimensional data to a polygonal visualization, aiding understanding of network decisions, learning dynamics, and stability, applicable to any black box system with continuous outputs.

Contribution

The paper presents a novel visualization technique that reveals neural network decision processes and dynamics, enhancing interpretability and analysis of black box models.

Findings

Visualization shows learning dynamics and decision boundaries.

Method helps compare different networks and detect potential issues.

Applicable to any black box system with continuous outputs.

Abstract

Neural networks are commonly regarded as black boxes performing incomprehensible functions. For classification problems networks provide maps from high dimensional feature space to K-dimensional image space. Images of training vector are projected on polygon vertices, providing visualization of network function. Such visualization may show the dynamics of learning, allow for comparison of different networks, display training vectors around which potential problems may arise, show differences due to regularization and optimization procedures, investigate stability of network classification under perturbation of original vectors, and place new data sample in relation to training data, allowing for estimation of confidence in classification of a given sample. An illustrative example for the three-class Wine data and five-class Satimage data is described. The visualization method proposed here is applicable to any black box system that provides continuous outputs.