GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems

TL;DR

GenTREC introduces a novel, low-cost test collection for evaluating information retrieval systems, generated entirely by large language models, reducing reliance on manual relevance judgments while maintaining evaluation reliability.

Contribution

This paper presents the first IR test collection created solely from LLM-generated documents, demonstrating its effectiveness and compatibility with traditional collections for system evaluation.

Findings

GenTREC's IR system rankings align with traditional collections for key metrics.

The collection contains nearly 97,000 documents and 19,000 relevance judgments.

The approach significantly reduces resource requirements for IR evaluation.

Abstract

Building test collections for Information Retrieval evaluation has traditionally been a resource-intensive and time-consuming task, primarily due to the dependence on manual relevance judgments. While various cost-effective strategies have been explored, the development of such collections remains a significant challenge. In this paper, we present GenTREC , the first test collection constructed entirely from documents generated by a Large Language Model (LLM), eliminating the need for manual relevance judgments. Our approach is based on the assumption that documents generated by an LLM are inherently relevant to the prompts used for their generation. Based on this heuristic, we utilized existing TREC search topics to generate documents. We consider a document relevant only to the prompt that generated it, while other document-topic pairs are treated as non-relevant. To introduce realistic retrieval challenges, we also generated non-relevant documents, ensuring that IR systems are tested against a diverse and robust set of materials. The resulting GenTREC collection comprises 96,196 documents, 300 topics, and 18,964 relevance "judgments". We conducted extensive experiments to evaluate GenTREC in terms of document quality, relevance judgment accuracy, and evaluation reliability. Notably, our findings indicate that the ranking of IR systems using GenTREC is compatible with the evaluations conducted using traditional TREC test collections, particularly for P@100, MAP, and RPrec metrics. Overall, our results show that our proposed approach offers a promising, low-cost alternative for IR evaluation, significantly reducing the burden of building and maintaining future IR evaluation resources.