Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators
Hyo Jin Do, Rachel Ostrand, Werner Geyer, Keerthiram Murugesan, Dennis Wei, Justin Weisz

TL;DR
This paper investigates how to effectively communicate factuality scores of LLM responses to users, finding that color-coded phrase indicators improve trust and validation ease, guiding better design of transparent AI systems.
Contribution
It introduces and empirically tests a visual design strategy for conveying factuality in LLM outputs, providing practical guidelines for enhancing transparency and user trust.
Findings
Color-coded phrase indicators increase user trust.
Participants find it easier to validate responses with visual cues.
Design guidelines improve LLM transparency and user experience.
Abstract
Large language models (LLMs) are susceptible to generating inaccurate or false information, often referred to as "hallucinations" or "confabulations." While several technical advancements have been made to detect hallucinated content by assessing the factuality of the model's responses, there is still limited research on how to effectively communicate this information to users. To address this gap, we conducted two scenario-based experiments with a total of 208 participants to systematically compare the effects of various design strategies for communicating factuality scores by assessing participants' ratings of trust, ease in validating response accuracy, and preference. Our findings reveal that participants preferred and trusted a design in which all phrases within a response were color-coded based on factuality scores. Participants also found it easier to validate accuracy of the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsArtificial Intelligence in Healthcare and Education · Ethics and Social Impacts of AI · Explainable Artificial Intelligence (XAI)
