Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search
Clemencia Siro, Zahra Abbasiantaeb, Yifei Yuan, Mohammad Aliannejadi, and Maarten de Rijke

TL;DR
This study investigates how images influence clarifying questions in conversational search, revealing that images enhance engagement and query precision depending on the task and user expertise, informing better multimodal system design.
Contribution
It provides the first comprehensive user study on the impact of images in clarifying questions within conversational search, highlighting task-dependent effects and user preferences.
Findings
Participants preferred multimodal questions for answering clarifications.
Images helped maintain engagement across user expertise levels.
Text-only questions led to more precise queries and better retrieval performance.
Abstract
Conversational search systems increasingly employ clarifying questions to refine user queries and improve the search experience. Previous studies have demonstrated the usefulness of text-based clarifying questions in enhancing both retrieval performance and user experience. While images have been shown to improve retrieval performance in various contexts, their impact on user performance when incorporated into clarifying questions remains largely unexplored. We conduct a user study with 73 participants to investigate the role of images in conversational search, specifically examining their effects on two search-related tasks: (i) answering clarifying questions and (ii) query reformulation. We compare the effect of multimodal and text-only clarifying questions in both tasks within a conversational search context from various perspectives. Our findings reveal that while participants…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInformation Retrieval and Search Behavior · Multimodal Machine Learning Applications · Expert finding and Q&A systems
