Seeing to Think? How Source Transparency Design Shapes Interactive Information Seeking and Evaluation in Conversational AI

TL;DR

This study examines how different source presentation interfaces in conversational AI influence user behavior, trust, and critical thinking, revealing trade-offs between seamless verification and reflective engagement.

Contribution

It provides empirical evidence on how specific interface designs impact information seeking, verification, and critical thinking in conversational AI systems.

Findings

Hover Card facilitates seamless verification during tasks.

Aligned Sidebar improves critical thinking with high citation density.

Designs influence exploration strategies and evidence integration.

Abstract

Conversational AI systems increasingly function as primary interfaces for information seeking, yet how they present sources to support information evaluation remains under-explored. This paper investigates how source transparency design shapes interactive information seeking, trust, and critical engagement. We conducted a controlled between-subjects experiment (N=372) comparing four source presentation interfaces - Collapsible, Hover Card, Footer, and Aligned Sidebar - varying in visibility and accessibility. Using fine-grained behavioral analysis and automated critical thinking assessment, we found that interface design fundamentally alters exploration strategies and evidence integration. While the Hover Card interface facilitated seamless, on-demand verification during the task, the Aligned Sidebar uniquely mitigated the negative effects of information overload: as citation density increased, Sidebar users demonstrated significantly higher critical thinking and synthesis scores compared to other conditions. Our results highlight a trade-off between designs that support workflow fluency and those that enforce reflective verification, offering practical implications for designing adaptive and responsible conversational AI that fosters critical engagement with AI generated content.