Machine Explanations and Human Understanding

TL;DR

This paper formalizes how explanations influence human understanding of machine learning models, emphasizing the role of human intuitions and providing a framework for improving explanations to better support human-AI decision making.

Contribution

It introduces a formal causal diagram framework to analyze the conditions under which explanations enhance human understanding, highlighting the importance of human intuitions.

Findings

Explanations can improve understanding of model decision boundary with assumptions about human intuitions.

Without assumptions, explanations cannot improve understanding of task decision boundary or model error.

Empirical studies confirm the critical role of human feature relevance intuitions in detecting model errors.

Abstract

Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even negative results. An open question, therefore, is under what conditions explanations can improve human understanding and in what way. Using adapted causal diagrams, we provide a formal characterization of the interplay between machine explanations and human understanding, and show how human intuitions play a central role in enabling human understanding. Specifically, we identify three core concepts of interest that cover all existing quantitative measures of understanding in the context of human-AI decision making: task decision boundary, model decision boundary, and model error. Our key result is that without assumptions about task-specific intuitions, explanations may potentially improve human understanding of model decision boundary, but they cannot improve human understanding of task decision boundary or model error. To achieve complementary human-AI performance, we articulate possible ways on how explanations need to work with human intuitions. For instance, human intuitions about the relevance of features (e.g., education is more important than age in predicting a person's income) can be critical in detecting model error. We validate the importance of human intuitions in shaping the outcome of machine explanations with empirical human-subject studies. Overall, our work provides a general framework along with actionable implications for future algorithmic development and empirical experiments of machine explanations.