Understanding Modality Preferences in Search Clarification

TL;DR

This paper investigates how different clarification question modalities, including text, images, and their combination, affect user preferences in search engines, introducing a new dataset and analyzing the effectiveness of automated image generation.

Contribution

It introduces the MIMICS-MM dataset for multi-modal search clarification and evaluates user preferences and the quality of model-generated images versus human-collected ones.

Findings

Users prefer multi-modal clarification over uni-modal approaches.

Stable Diffusion effectively generates relevant clarification images.

Model-generated images are comparable in quality and relevance to human-collected images.

Abstract

This study is the first attempt to explore the impact of clarification question modality on user preference in search engines. We introduce the multi-modal search clarification dataset, MIMICS-MM, containing clarification questions with associated expert-collected and model-generated images. We analyse user preferences over different clarification modes of text, image, and combination of both through crowdsourcing by taking into account image and text quality, clarity, and relevance. Our findings demonstrate that users generally prefer multi-modal clarification over uni-modal approaches. We explore the use of automated image generation techniques and compare the quality, relevance, and user preference of model-generated images with human-collected ones. The study reveals that text-to-image generation models, such as Stable Diffusion, can effectively generate multi-modal clarification questions. By investigating multi-modal clarification, this research establishes a foundation for future advancements in search systems.