How model accuracy and explanation fidelity influence user trust

TL;DR

This study investigates how model accuracy and explanation fidelity affect user trust in AI systems, revealing that accuracy is more crucial for trust than explanations, which can sometimes harm trust if untruthful.

Contribution

It provides empirical evidence that accuracy outweighs explanation fidelity in building user trust and highlights potential pitfalls of explanations in AI systems.

Findings

Accuracy has a greater impact on trust than explanation fidelity.

Untruthful explanations can decrease user trust.

High-fidelity explanations do not necessarily increase trust in poor models.

Abstract

Machine learning systems have become popular in fields such as marketing, financing, or data mining. While they are highly accurate, complex machine learning systems pose challenges for engineers and users. Their inherent complexity makes it impossible to easily judge their fairness and the correctness of statistically learned relations between variables and classes. Explainable AI aims to solve this challenge by modelling explanations alongside with the classifiers, potentially improving user trust and acceptance. However, users should not be fooled by persuasive, yet untruthful explanations. We therefore conduct a user study in which we investigate the effects of model accuracy and explanation fidelity, i.e. how truthfully the explanation represents the underlying model, on user trust. Our findings show that accuracy is more important for user trust than explainability. Adding an explanation for a classification result can potentially harm trust, e.g. when adding nonsensical explanations. We also found that users cannot be tricked by high-fidelity explanations into having trust for a bad classifier. Furthermore, we found a mismatch between observed (implicit) and self-reported (explicit) trust.