Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search

TL;DR

This study investigates how images influence clarifying questions in conversational search, revealing that images enhance engagement and query precision depending on the task and user expertise, informing better multimodal system design.

Contribution

It provides the first comprehensive user study on the impact of images in clarifying questions within conversational search, highlighting task-dependent effects and user preferences.

Findings

Participants preferred multimodal questions for answering clarifications.

Images helped maintain engagement across user expertise levels.

Text-only questions led to more precise queries and better retrieval performance.

Abstract

Conversational search systems increasingly employ clarifying questions to refine user queries and improve the search experience. Previous studies have demonstrated the usefulness of text-based clarifying questions in enhancing both retrieval performance and user experience. While images have been shown to improve retrieval performance in various contexts, their impact on user performance when incorporated into clarifying questions remains largely unexplored. We conduct a user study with 73 participants to investigate the role of images in conversational search, specifically examining their effects on two search-related tasks: (i) answering clarifying questions and (ii) query reformulation. We compare the effect of multimodal and text-only clarifying questions in both tasks within a conversational search context from various perspectives. Our findings reveal that while participants showed a strong preference for multimodal questions when answering clarifying questions, preferences were more balanced in the query reformulation task. The impact of images varied with both task type and user expertise. In answering clarifying questions, images helped maintain engagement across different expertise levels, while in query reformulation they led to more precise queries and improved retrieval performance. Interestingly, for clarifying question answering, text-only setups demonstrated better user performance as they provided more comprehensive textual information in the absence of images. These results provide valuable insights for designing effective multimodal conversational search systems, highlighting that the benefits of visual augmentation are task-dependent and should be strategically implemented based on the specific search context and user characteristics.