On Evaluating Explanation Utility for Human-AI Decision Making in NLP

TL;DR

This paper critically evaluates the effectiveness of explanations in human-AI decision-making within NLP, emphasizing the need for application-grounded assessments, dataset selection criteria, and rethinking human-AI teaming strategies.

Contribution

It reviews existing evaluation metrics, establishes dataset selection criteria, and demonstrates the limited impact of explanations on decision speed and accuracy in NLP tasks.

Findings

Only 4 out of 50 datasets meet the criteria for evaluation

Explanations do not speed up decision-making without accuracy loss

Revisiting human-AI teaming and automatic deferral is necessary

Abstract

Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for. More human-centered, application-grounded evaluations of explanations are needed to settle this. Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies. To aid with this, we first review existing metrics suitable for application-grounded evaluation. We then establish criteria to select appropriate datasets, and using them, we find that only 4 out of over 50 datasets available for explainability research in NLP meet them. We then demonstrate the importance of reassessing the state of the art to form and study human-AI teams: teaming people with models for certain tasks might only now start to make sense, and for others, it remains unsound. Finally, we present the exemplar studies of human-AI decision-making for one of the identified tasks -- verifying the correctness of a legal claim given a contract. Our results show that providing AI predictions, with or without explanations, does not cause decision makers to speed up their work without compromising performance. We argue for revisiting the setup of human-AI teams and improving automatic deferral of instances to AI, where explanations could play a useful role.