Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset

TL;DR

This study evaluates whether different explanation methods improve user understanding of image classification models, finding that simple baselines can outperform more complex explanations in helping users identify relevant attributes.

Contribution

The paper introduces a synthetic dataset generator and conducts a user study comparing explanation techniques, highlighting the importance of user-centered evaluation of interpretability methods.

Findings

Baseline explanations outperform concept-based explanations.

Counterfactual explanations help users identify some attributes more accurately.

Emphasizes measuring user reasoning about model biases rather than just technical metrics.

Abstract

A variety of methods exist to explain image classification models. However, whether they provide any benefit to users over simply comparing various inputs and the model's respective predictions remains unclear. We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations. To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model. In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth. Our results show that the baseline outperformed concept-based explanations. Counterfactual explanations from an invertible neural network performed similarly as the baseline. Still, they allowed users to identify some attributes more accurately. Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks. We open-source our study and dataset so it can serve as a blue-print for future studies. For code see, https://github.com/berleon/do_users_benefit_from_interpretable_vision