Prototype Generation: Robust Feature Visualisation for Data Independent Interpretability

TL;DR

This paper presents Prototype Generation, a robust feature visualization method for model-agnostic interpretability that produces natural activation paths and reveals model biases and spurious correlations.

Contribution

It introduces a new prototype generation technique that enhances the trustworthiness of feature visualizations for understanding image classification models.

Findings

Generated prototypes produce natural activation paths.

Quantitative similarity measures validate prototype authenticity.

Interpretations reveal biases and spurious correlations.

Abstract

We introduce Prototype Generation, a stricter and more robust form of feature visualisation for model-agnostic, data-independent interpretability of image classification models. We demonstrate its ability to generate inputs that result in natural activation paths, countering previous claims that feature visualisation algorithms are untrustworthy due to the unnatural internal activations. We substantiate these claims by quantitatively measuring similarity between the internal activations of our generated prototypes and natural images. We also demonstrate how the interpretation of generated prototypes yields important insights, highlighting spurious correlations and biases learned by models which quantitative methods over test-sets cannot identify.