Won't you see my neighbor?: User predictions, mental models, and similarity-based explanations of AI classifiers

TL;DR

This study investigates how providing humans with similarity-based explanations of AI classifiers improves their ability to predict AI errors and influences their mental models, highlighting the importance of interpretability in AI-human collaboration.

Contribution

The paper demonstrates that similarity-based explanations enhance human prediction accuracy of AI classifications and affect mental models, introducing a novel approach to interpretability.

Findings

Providing neighbor information improves prediction accuracy

Similarity to neighbor images correlates with performance gains

Explanations influence human classification judgments

Abstract

Humans should be able work more effectively with artificial intelligence-based systems when they can predict likely failures and form useful mental models of how the systems work. We conducted a study of human's mental models of artificial intelligence systems using a high-performing image classifier, focusing on participants' ability to predict the classification result for a particular image. Participants viewed individual labeled images in one of two classes and then tried to predict whether the classifier would label them correctly. In this experiment we explored the effect of giving participants additional information about an image's nearest neighbors in a space representing the otherwise uninterpretable features extracted by the lower layers of the classifier's neural network. We found that providing this information did increase participants' prediction performance, and that the performance improvement could be related to the neighbor images' similarity to the target image. We also found indications that the presentation of this information may influence people's own classification of the target image -- that is, rather than just anthropomorphizing the system, in some cases the humans become "mechanomorphized" in their judgements.