Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated

TL;DR

This paper explores how psychophysical experiments can effectively evaluate the quality of interpretable machine learning explanations, revealing that automated metrics alone do not reflect human usefulness.

Contribution

It introduces psychophysical methods for assessing interpretability quality and demonstrates their superiority over traditional automated metrics in human-centered evaluation.

Findings

Automated metrics do not correlate well with human usefulness.

Psychophysical experiments provide robust interpretability assessment.

Human-in-the-loop evaluation is essential for authentic interpretability measurement.

Abstract

The field explainable artificial intelligence (XAI) has brought about an arsenal of methods to render Machine Learning (ML) predictions more interpretable. But how useful explanations provided by transparent ML methods are for humans remains difficult to assess. Here we investigate the quality of interpretable computer vision algorithms using techniques from psychophysics. In crowdsourced annotation tasks we study the impact of different interpretability approaches on annotation accuracy and task time. We compare these quality metrics with classical XAI, automated quality metrics. Our results demonstrate that psychophysical experiments allow for robust quality assessment of transparency in machine learning. Interestingly the quality metrics computed without humans in the loop did not provide a consistent ranking of interpretability methods nor were they representative for how useful an explanation was for humans. These findings highlight the potential of methods from classical psychophysics for modern machine learning applications. We hope that our results provide convincing arguments for evaluating interpretability in its natural habitat, human-ML interaction, if the goal is to obtain an authentic assessment of interpretability.