Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies

TL;DR

This study investigates how explanations, sources, and inconsistencies in LLM responses influence user reliance, aiming to promote appropriate trust and reduce overreliance on potentially incorrect outputs.

Contribution

It identifies key response features affecting reliance and demonstrates how sources and inconsistencies can mitigate overtrust in LLM outputs.

Findings

Explanations increase reliance on both correct and incorrect answers.

Providing sources reduces reliance on incorrect responses.

Inconsistencies in explanations decrease overreliance on wrong answers.

Abstract

Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct. Mitigating such overreliance is a key challenge. Through a think-aloud study in which participants use an LLM-infused application to answer objective questions, we identify several features of LLM responses that shape users' reliance: explanations (supporting details for answers), inconsistencies in explanations, and sources. Through a large-scale, pre-registered, controlled experiment (N=308), we isolate and study the effects of these features on users' reliance, accuracy, and other measures. We find that the presence of explanations increases reliance on both correct and incorrect responses. However, we observe less reliance on incorrect responses when sources are provided or when explanations exhibit inconsistencies. We discuss the implications of these findings for fostering appropriate reliance on LLMs.