From Explainable to Explained AI: Ideas for Falsifying and Quantifying Explanations

TL;DR

This paper introduces a human-machine interaction system that tests and quantifies explanations of deep learning models in digital pathology, enabling validation and comparison of different explanations for medical image classifiers.

Contribution

It presents a novel system combining slide viewers, visual experiments, and vision-language models to test and quantify explanations in AI-based pathology diagnostics.

Findings

Allows qualitative testing of explanation claims

Quantifies the predictiveness of explanations

Distinguishes between competing explanations

Abstract

Explaining deep learning models is essential for clinical integration of medical image analysis systems. A good explanation highlights if a model depends on spurious features that undermines generalization and harms a subset of patients or, conversely, may present novel biological insights. Although techniques like GradCAM can identify influential features, they are measurement tools that do not themselves form an explanation. We propose a human-machine-VLM interaction system tailored to explaining classifiers in computational pathology, including multi-instance learning for whole-slide images. Our proof of concept comprises (1) an AI-integrated slide viewer to run sliding-window experiments to test claims of an explanation, and (2) quantification of an explanation's predictiveness using general-purpose vision-language models. The results demonstrate that this allows us to qualitatively test claims of explanations and can quantifiably distinguish competing explanations. This offers a practical path from explainable AI to explained AI in digital pathology and beyond. Code and prompts are available at https://github.com/nki-ai/x2x.