Sampling Prediction-Matching Examples in Neural Networks: A Probabilistic Programming Approach

TL;DR

This paper introduces a probabilistic programming method for sampling examples at specific prediction confidence levels in neural networks, aiding interpretability without requiring differentiability.

Contribution

It presents a novel sampling approach to explore classifier prediction level sets, applicable to arbitrary models including neural networks, enhancing interpretability and understanding.

Findings

Effective sampling of prediction level sets demonstrated on synthetic data.

Applied method successfully to MNIST, illustrating practical utility.

Method does not require classifier differentiability.

Abstract

Though neural network models demonstrate impressive performance, we do not understand exactly how these black-box models make individual predictions. This drawback has led to substantial research devoted to understand these models in areas such as robustness, interpretability, and generalization ability. In this paper, we consider the problem of exploring the prediction level sets of a classifier using probabilistic programming. We define a prediction level set to be the set of examples for which the predictor has the same specified prediction confidence with respect to some arbitrary data distribution. Notably, our sampling-based method does not require the classifier to be differentiable, making it compatible with arbitrary classifiers. As a specific instantiation, if we take the classifier to be a neural network and the data distribution to be that of the training data, we can obtain examples that will result in specified predictions by the neural network. We demonstrate this technique with experiments on a synthetic dataset and MNIST. Such level sets in classification may facilitate human understanding of classification behaviors.