A Case Study on Designing Evaluations of ML Explanations with Simulated User Studies

TL;DR

This paper investigates the use of simulated user evaluations (SimEvals) as a cost-effective method to assess ML explanations in e-commerce fraud detection, finding that SimEvals can replicate key results of real user studies and aid in designing evaluations.

Contribution

First application of SimEvals on a real-world use case demonstrating their potential to replicate user study findings and inform evaluation design.

Findings

SimEvals suggested all explainers performed equally, matching user study results.

SimEvals indicated explanations did not outperform baseline models.

Results support using SimEvals as a preliminary evaluation step.

Abstract

When conducting user studies to ascertain the usefulness of model explanations in aiding human decision-making, it is important to use real-world use cases, data, and users. However, this process can be resource-intensive, allowing only a limited number of explanation methods to be evaluated. Simulated user evaluations (SimEvals), which use machine learning models as a proxy for human users, have been proposed as an intermediate step to select promising explanation methods. In this work, we conduct the first SimEvals on a real-world use case to evaluate whether explanations can better support ML-assisted decision-making in e-commerce fraud detection. We study whether SimEvals can corroborate findings from a user study conducted in this fraud detection context. In particular, we find that SimEvals suggest that all considered explainers are equally performant, and none beat a baseline without explanations -- this matches the conclusions of the original user study. Such correspondences between our results and the original user study provide initial evidence in favor of using SimEvals before running user studies. We also explore the use of SimEvals as a cheap proxy to explore an alternative user study set-up. We hope that this work motivates further study of when and how SimEvals should be used to aid in the design of real-world evaluations.