Explaining Chest X-ray Pathology Models using Textual Concepts

TL;DR

This paper introduces CoCoX, a method that uses vision-language models and textual radiology reports to generate human-understandable explanations for chest X-ray diagnoses without requiring annotated datasets.

Contribution

It presents a novel approach leveraging pre-trained vision-language models and textual concepts to explain chest X-ray pathology models, bypassing the need for manual concept annotations.

Findings

Generated explanations are semantically meaningful.

Explanations are faithful to underlying pathologies.

Method effectively explains multiple cardiothoracic conditions.

Abstract

Deep learning models have revolutionized medical imaging and diagnostics, yet their opaque nature poses challenges for clinical adoption and trust. Amongst approaches to improve model interpretability, concept-based explanations aim to provide concise and human-understandable explanations of any arbitrary classifier. However, such methods usually require a large amount of manually collected data with concept annotation, which is often scarce in the medical domain. In this paper, we propose Conceptual Counterfactual Explanations for Chest X-ray (CoCoX), which leverages the joint embedding space of an existing vision-language model (VLM) to explain black-box classifier outcomes without the need for annotated datasets. Specifically, we utilize textual concepts derived from chest radiography reports and a pre-trained chest radiography-based VLM to explain three common cardiothoracic pathologies. We demonstrate that the explanations generated by our method are semantically meaningful and faithful to underlying pathologies.