SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
Hossein A. Rahmani, Xi Wang, Emine Yilmaz, Nick Craswell, Bhaskar, Mitra, Paul Thomas

TL;DR
This paper introduces SynDL, a large-scale synthetic test collection for passage retrieval, created using language models to generate relevance labels, enabling scalable evaluation of IR systems.
Contribution
The paper presents a novel large-scale synthetic dataset for passage retrieval, extending the TREC Deep Learning collection with LLM-generated relevance judgments.
Findings
High correlation between system rankings using synthetic and human labels
Enables scalable and cost-effective IR system evaluation
Supports large-scale research without extensive human annotation
Abstract
Large-scale test collections play a crucial role in Information Retrieval (IR) research. However, according to the Cranfield paradigm and the research into publicly available datasets, the existing information retrieval research studies are commonly developed on small-scale datasets that rely on human assessors for relevance judgments - a time-intensive and expensive process. Recent studies have shown the strong capability of Large Language Models (LLMs) in producing reliable relevance judgments with human accuracy but at a greatly reduced cost. In this paper, to address the missing large-scale ad-hoc document retrieval dataset, we extend the TREC Deep Learning Track (DL) test collection via additional language model synthetic labels to enable researchers to test and evaluate their search systems at a large scale. Specifically, such a test collection includes more than 1,900 test…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Speech Recognition and Synthesis
