Demographically-Inspired Query Variants Using an LLM

TL;DR

This paper introduces a method using an LLM to generate demographically-inspired query variants, enhancing test collection diversity and providing new insights into IR system evaluation across different user profiles.

Contribution

It presents a novel approach to create user profile-specific query variants with an LLM, improving test collection diversity and evaluation insights.

Findings

Query variants influence system ranking outcomes.

User profiles significantly affect perceived system effectiveness.

LLM-generated variants reflect real user diversity.

Abstract

This study proposes a method to diversify queries in existing test collections to reflect some of the diversity of search engine users, aligning with an earlier vision of an 'ideal' test collection. A Large Language Model (LLM) is used to create query variants: alternative queries that have the same meaning as the original. These variants represent user profiles characterised by different properties, such as language and domain proficiency, which are known in the IR literature to influence query formulation. The LLM's ability to generate query variants that align with user profiles is empirically validated, and the variants' utility is further explored for IR system evaluation. Results demonstrate that the variants impact how systems are ranked and show that user profiles experience significantly different levels of system effectiveness. This method enables an alternative perspective on system evaluation where we can observe both the impact of user profiles on system rankings and how system performance varies across users.