Visualizing Neural Network Imagination

TL;DR

This paper introduces a method to visualize and interpret the hidden states of neural networks, demonstrating high interpretability on simple tasks using novel visualization, autoencoder, and adversarial techniques.

Contribution

It presents a new approach combining visualization and interpretability metrics to understand neural network hidden states, with techniques to enhance interpretability.

Findings

Hidden states can be highly interpretable on simple tasks

Autoencoder and adversarial methods improve interpretability

Quantitative interpretability metric validates visualizations

Abstract

In certain situations, neural networks will represent environment states in their hidden activations. Our goal is to visualize what environment states the networks are representing. We experiment with a recurrent neural network (RNN) architecture with a decoder network at the end. After training, we apply the decoder to the intermediate representations of the network to visualize what they represent. We define a quantitative interpretability metric and use it to demonstrate that hidden states can be highly interpretable on a simple task. We also develop autoencoder and adversarial techniques and show that benefit interpretability.