MIMICS-Duo: Offline & Online Evaluation of Search Clarification

TL;DR

MIMICS-Duo is a new dataset that facilitates comprehensive offline and online evaluation of search clarification methods, addressing resource limitations in this research area.

Contribution

It introduces MIMICS-Duo, a large, annotated dataset enabling multi-dimensional evaluation of search clarification techniques, including offline and online assessments.

Findings

Demonstrates the relationship between offline and online clarification datasets

Provides extensive analysis of search clarification data

Outlines future research directions using MIMICS-Duo

Abstract

Asking clarification questions is an active area of research; however, resources for training and evaluating search clarification methods are not sufficient. To address this issue, we describe MIMICS-Duo, a new freely available dataset of 306 search queries with multiple clarifications (a total of 1,034 query-clarification pairs). MIMICS-Duo contains fine-grained annotations on clarification questions and their candidate answers and enhances the existing MIMICS datasets by enabling multi-dimensional evaluation of search clarification methods, including online and offline evaluation. We conduct extensive analysis to demonstrate the relationship between offline and online search clarification datasets and outline several research directions enabled by MIMICS-Duo. We believe that this resource will help researchers better understand clarification in search.