Hide or Highlight: Understanding the Impact of Factuality Expression on User Trust

TL;DR

This study investigates how different methods of communicating factuality in AI outputs affect user trust, finding that hiding or vague disclosures increase trust without reducing perceived quality.

Contribution

It introduces and empirically tests four novel strategies for disclosing factuality in AI outputs and compares their impact on user trust in a question-answering context.

Findings

Opaque and ambiguity strategies increase user trust.

Hiding less factual content maintains perceived answer quality.

Different disclosure methods significantly influence trust levels.

Abstract

Large language models are known to produce outputs that are plausible but factually incorrect. To prevent people from making erroneous decisions by blindly trusting AI, researchers have explored various ways of communicating factuality estimates in AI-generated outputs to end-users. However, little is known about whether revealing content estimated to be factually incorrect influences users' trust when compared to hiding it altogether. We tested four different ways of disclosing an AI-generated output with factuality assessments: transparent (highlights less factual content), attention (highlights factual content), opaque (removes less factual content), ambiguity (makes less factual content vague), and compared them with a baseline response without factuality information. We conducted a human subjects research (N = 148) using the strategies in question-answering scenarios. We found that the opaque and ambiguity strategies led to higher trust while maintaining perceived answer quality, compared to the other strategies. We discuss the efficacy of hiding presumably less factual content to build end-user trust.