Understanding Mental Models of Generative Conversational Search and The Effect of Interface Transparency

TL;DR

This study explores how users understand generative conversational search systems and how interface transparency affects their mental models, revealing that abstract mental models can hinder trust and suggesting hybrid search interfaces as a future direction.

Contribution

It provides empirical insights into users' mental models of generative conversational search and highlights the importance of transparency and hybrid interfaces for improved design.

Findings

Most mental models were too abstract for explaining individual search instances.

Abstract mental models may hinder appropriate trust in conversational search.

Hybrid web-conversational search is a promising future interface direction.

Abstract

The experience and adoption of conversational search is tied to the accuracy and completeness of users' mental models -- their internal frameworks for understanding and predicting system behaviour. Thus, understanding these models can reveal areas for design interventions. Transparency is one such intervention which can improve system interpretability and enable mental model alignment. While past research has explored mental models of search engines, those of generative conversational search remain underexplored, even while the popularity of these systems soars. To address this, we conducted a study with 16 participants, who performed 4 search tasks using 4 conversational interfaces of varying transparency levels. Our analysis revealed that most user mental models were too abstract to support users in explaining individual search instances. These results suggest that 1) mental models may pose a barrier to appropriate trust in conversational search, and 2) hybrid web-conversational search is a promising novel direction for future search interface design.