Beliefs and Misconceptions around Integrated Conversational AI

TL;DR

This study investigates how users perceive and trust integrated conversational AI in web browsers, revealing reliance on existing beliefs and citation cues affecting trust and fact-checking behaviors.

Contribution

It provides empirical insights into user beliefs, trust, and prompting strategies when interacting with integrated conversational AI like Copilot in a web browser.

Findings

Participants relied on existing perceptions of LLMs and search engines.

Citations increased trustworthiness without prompting verification.

Users often checked the same sources as the AI for fact-checking.

Abstract

LLM-driven conversational AI is beginning to disappear into the background, shifting from something used directly towards something increasingly integrated into existing workflows. In the process, markers of origin and training are smoothed away as LLMs become commodified in the eyes of users. We explore how people approach using a web browser with conversational AI built in, focusing on how they develop their understanding and determine whether to trust its outputs. We conducted a study where 20 participants used the Copilot AI features in Microsoft Edge to conduct information retrieval and planning tasks. Participants relied on a combination of existing perceptions of LLMs and internet search, tracing the effect of beliefs about how Copilot generated answers on prompting strategies. The inclusion of citations increased the trustworthiness of answers without participants feeling the need to be check them, with participants often reaching for the same information sources as the CAI when fact-checking.