Tip of the Tongue Query Elicitation for Simulated Evaluation
Yifan He, To Eun Kim, Fernando Diaz, Jaime Arguello, Bhaskar Mitra

TL;DR
This paper introduces methods using large language models and human participants to generate and collect Tip-of-the-Tongue queries, enabling more scalable and domain-diverse evaluation of TOT retrieval systems.
Contribution
It presents novel approaches for eliciting TOT queries via LLMs and visual stimuli, reducing reliance on CQA data and broadening domain coverage for evaluation.
Findings
LLM-based synthetic TOT queries correlate well with CQA queries in ranking tasks.
Human-elicited queries closely resemble CQA queries in linguistic features.
The methods enable scalable, domain-diverse TOT query collection for evaluation.
Abstract
Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scenarios. Research on TOT retrieval is further constrained by the challenge of collecting queries, as current approaches rely heavily on community question-answering (CQA) websites, leading to labor-intensive evaluation and domain bias. To overcome these limitations, we introduce two methods for eliciting TOT queries - leveraging large language models (LLMs) and human participants - to facilitate simulated evaluations of TOT retrieval systems. Our LLM-based TOT user simulator generates synthetic TOT queries at scale, achieving high correlations with how CQA-based TOT queries rank TOT retrieval systems when tested in the Movie domain. Additionally, these synthetic queries exhibit high linguistic…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsLinguistics and Cultural Studies · Educational Technology and Pedagogy · Voice and Speech Disorders
