SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval

TL;DR

This paper introduces SynDL, a large-scale synthetic test collection for passage retrieval, created using language models to generate relevance labels, enabling scalable evaluation of IR systems.

Contribution

The paper presents a novel large-scale synthetic dataset for passage retrieval, extending the TREC Deep Learning collection with LLM-generated relevance judgments.

Findings

High correlation between system rankings using synthetic and human labels

Enables scalable and cost-effective IR system evaluation

Supports large-scale research without extensive human annotation

Abstract

Large-scale test collections play a crucial role in Information Retrieval (IR) research. However, according to the Cranfield paradigm and the research into publicly available datasets, the existing information retrieval research studies are commonly developed on small-scale datasets that rely on human assessors for relevance judgments - a time-intensive and expensive process. Recent studies have shown the strong capability of Large Language Models (LLMs) in producing reliable relevance judgments with human accuracy but at a greatly reduced cost. In this paper, to address the missing large-scale ad-hoc document retrieval dataset, we extend the TREC Deep Learning Track (DL) test collection via additional language model synthetic labels to enable researchers to test and evaluate their search systems at a large scale. Specifically, such a test collection includes more than 1,900 test queries from the previous years of tracks. We compare system evaluation with past human labels from past years and find that our synthetically created large-scale test collection can lead to highly correlated system rankings.