Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information

TL;DR

This paper introduces a framework that interprets neural network task solving by quantifying and synthesizing previously unused information, progressively refining understanding with new experience and providing visual explanations.

Contribution

It proposes a novel method to synthesize task interpretations by quantifying un-conceptualized information and its utilization in neural networks, using the variational information bottleneck.

Findings

Framework can identify and quantify un-conceptualized information.

It progressively refines interpretation with new tasks.

Provides visual explanations of used information.

Abstract

A deep neural network is a good task solver, but it is difficult to make sense of its operation. People have different ideas about how to form the interpretation about its operation. We look at this problem from a new perspective where the interpretation of task solving is synthesized by quantifying how much and what previously unused information is exploited in addition to the information used to solve previous tasks. First, after learning several tasks, the network acquires several information partitions related to each task. We propose that the network, then, learns the minimal information partition that supplements previously learned information partitions to more accurately represent the input. This extra partition is associated with un-conceptualized information that has not been used in previous tasks. We manage to identify what un-conceptualized information is used and quantify the amount. To interpret how the network solves a new task, we quantify as meta-information how much information from each partition is extracted. We implement this framework with the variational information bottleneck technique. We test the framework with the MNIST and the CLEVR dataset. The framework is shown to be able to compose information partitions and synthesize experience-dependent interpretation in the form of meta-information. This system progressively improves the resolution of interpretation upon new experience by converting a part of the un-conceptualized information partition to a task-related partition. It can also provide a visual interpretation by imaging what is the part of previously un-conceptualized information that is needed to solve a new task.