# Concept-Centric Visual Turing Tests for Method Validation

## TL;DR

This paper introduces a concept-centric evaluation framework for medical image classification methods, inspired by the Turing Test, to better understand how models interpret key clinical concepts beyond traditional metrics.

## Contribution

It proposes a novel interpretability assessment using a Twenty Questions paradigm and probabilistic modeling to evaluate medical imaging methods' understanding of clinical concepts.

## Key findings

- The probabilistic model reveals dataset and method biases.
- The approach reduces the number of queries needed for evaluation.
- It provides deeper insights into model interpretability in clinical tasks.

## Abstract

Recent advances in machine learning for medical imaging have led to impressive increases in model complexity and overall capabilities. However, the ability to discern the precise information a machine learning method is using to make decisions has lagged behind and it is often unclear how these performances are in fact achieved. Conventional evaluation metrics that reduce method performance to a single number or a curve only provide limited insights. Yet, systems used in clinical practice demand thorough validation that such crude characterizations miss. To this end, we present a framework to evaluate classification methods based on a number of interpretable concepts that are crucial for a clinical task. Our approach is inspired by the Turing Test concept and how to devise a test that adaptively questions a method for its ability to interpret medical images. To do this, we make use of a Twenty Questions paradigm whereby we use a probabilistic model to characterize the method's capacity to grasp task-specific concepts, and we introduce a strategy to sequentially query the method according to its previous answers. The results show that the probabilistic model is able to expose both the dataset's and the method's biases, and can be used to reduced the number of queries needed for confident performance evaluation.