Tip of the Tongue Query Elicitation for Simulated Evaluation

TL;DR

This paper introduces methods using large language models and human participants to generate and collect Tip-of-the-Tongue queries, enabling more scalable and domain-diverse evaluation of TOT retrieval systems.

Contribution

It presents novel approaches for eliciting TOT queries via LLMs and visual stimuli, reducing reliance on CQA data and broadening domain coverage for evaluation.

Findings

LLM-based synthetic TOT queries correlate well with CQA queries in ranking tasks.

Human-elicited queries closely resemble CQA queries in linguistic features.

The methods enable scalable, domain-diverse TOT query collection for evaluation.

Abstract

Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scenarios. Research on TOT retrieval is further constrained by the challenge of collecting queries, as current approaches rely heavily on community question-answering (CQA) websites, leading to labor-intensive evaluation and domain bias. To overcome these limitations, we introduce two methods for eliciting TOT queries - leveraging large language models (LLMs) and human participants - to facilitate simulated evaluations of TOT retrieval systems. Our LLM-based TOT user simulator generates synthetic TOT queries at scale, achieving high correlations with how CQA-based TOT queries rank TOT retrieval systems when tested in the Movie domain. Additionally, these synthetic queries exhibit high linguistic similarity to CQA-derived queries. For human-elicited queries, we developed an interface that uses visual stimuli to place participants in a TOT state, enabling the collection of natural queries. In the Movie domain, system rank correlation and linguistic similarity analyses confirm that human-elicited queries are both effective and closely resemble CQA-based queries. These approaches reduce reliance on CQA-based data collection while expanding coverage to underrepresented domains, such as Landmark and Person. LLM-elicited queries for the Movie, Landmark, and Person domains have been released as test queries in the TREC 2024 TOT track, with human-elicited queries scheduled for inclusion in the TREC 2025 TOT track. Additionally, we provide source code for synthetic query generation and the human query collection interface, along with curated visual stimuli used for eliciting TOT queries.