Why Would You Suggest That? Human Trust in Language Model Responses

TL;DR

This paper investigates how explanations influence human trust in language models, revealing that explanations boost trust only when responses are compared directly, highlighting complexities in human-AI trust dynamics.

Contribution

It provides empirical evidence on the impact of explanations and response presentation on human trust in LLMs, emphasizing the importance of context and explanation faithfulness.

Findings

Explanations increase trust when responses are compared.

Trust gains vanish when responses are shown independently.

Humans trust all responses equally in isolation.

Abstract

The emergence of Large Language Models (LLMs) has revealed a growing need for human-AI collaboration, especially in creative decision-making scenarios where trust and reliance are paramount. Through human studies and model evaluations on the open-ended News Headline Generation task from the LaMP benchmark, we analyze how the framing and presence of explanations affect user trust and model performance. Overall, we provide evidence that adding an explanation in the model response to justify its reasoning significantly increases self-reported user trust in the model when the user has the opportunity to compare various responses. Position and faithfulness of these explanations are also important factors. However, these gains disappear when users are shown responses independently, suggesting that humans trust all model responses, including deceptive ones, equitably when they are shown in isolation. Our findings urge future research to delve deeper into the nuanced evaluation of trust in human-machine teaming systems.