Data Fusion of Synthetic Query Variants With Generative Large Language Models

TL;DR

This paper investigates using instruction-tuned Large Language Models to generate synthetic query variants for data fusion in information retrieval, demonstrating improved effectiveness over traditional methods and sharing resources publicly.

Contribution

It introduces a cost-efficient, unsupervised approach leveraging LLMs for synthetic query generation and data fusion, outperforming baseline and pseudo-relevance feedback methods.

Findings

Synthetic queries generated with context improve retrieval effectiveness.

Data fusion with synthetic queries outperforms single-query baselines.

The approach is lightweight, unsupervised, and cost-efficient.

Abstract

Considering query variance in information retrieval (IR) experiments is beneficial for retrieval effectiveness. Especially ranking ensembles based on different topically related queries retrieve better results than rankings based on a single query alone. Recently, generative instruction-tuned Large Language Models (LLMs) improved on a variety of different tasks in capturing human language. To this end, this work explores the feasibility of using synthetic query variants generated by instruction-tuned LLMs in data fusion experiments. More specifically, we introduce a lightweight, unsupervised, and cost-efficient approach that exploits principled prompting and data fusion techniques. In our experiments, LLMs produce more effective queries when provided with additional context information on the topic. Furthermore, our analysis based on four TREC newswire benchmarks shows that data fusion based on synthetic query variants is significantly better than baselines with single queries and also outperforms pseudo-relevance feedback methods. We publicly share the code and query datasets with the community as resources for follow-up studies.