Explainability's Gain is Optimality's Loss? -- How Explanations Bias Decision-making

TL;DR

This paper investigates how feature-based explanations in machine learning can introduce confirmation bias and bias decision outcomes, potentially leading to sub-optimal organizational decisions.

Contribution

It provides empirical evidence that explanations can bias decision-makers, highlighting the trade-off between explainability and optimality in machine learning applications.

Findings

Explanations induce confirmation bias in decision-makers.

Biases from explanations affect confidence and decision quality.

Field experiment demonstrates real-world impact of explanation-induced biases.

Abstract

Decisions in organizations are about evaluating alternatives and choosing the one that would best serve organizational goals. To the extent that the evaluation of alternatives could be formulated as a predictive task with appropriate metrics, machine learning algorithms are increasingly being used to improve the efficiency of the process. Explanations help to facilitate communication between the algorithm and the human decision-maker, making it easier for the latter to interpret and make decisions on the basis of predictions by the former. Feature-based explanations' semantics of causal models, however, induce leakage from the decision-maker's prior beliefs. Our findings from a field experiment demonstrate empirically how this leads to confirmation bias and disparate impact on the decision-maker's confidence in the predictions. Such differences can lead to sub-optimal and biased decision outcomes.