Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard

TL;DR

This paper introduces resources for the BEIR benchmark, including reproducible reference implementations of retrieval models and an official leaderboard, to enhance reproducibility, comparability, and research in zero-shot information retrieval across diverse domains.

Contribution

It provides reproducible reference implementations for dense and sparse retrieval models and establishes an official BEIR leaderboard for consistent model evaluation.

Findings

Reproducible implementations ease entry for new researchers.

The leaderboard enables fair comparison of retrieval models.

Facilitates future research in domain-specific information retrieval.

Abstract

BEIR is a benchmark dataset for zero-shot evaluation of information retrieval models across 18 different domain/task combinations. In recent years, we have witnessed the growing popularity of a representation learning approach to building retrieval models, typically using pretrained transformers in a supervised setting. This naturally begs the question: How effective are these models when presented with queries and documents that differ from the training data? Examples include searching in different domains (e.g., medical or legal text) and with different types of queries (e.g., keywords vs. well-formed questions). While BEIR was designed to answer these questions, our work addresses two shortcomings that prevent the benchmark from achieving its full potential: First, the sophistication of modern neural methods and the complexity of current software infrastructure create barriers to entry for newcomers. To this end, we provide reproducible reference implementations that cover the two main classes of approaches: learned dense and sparse models. Second, there does not exist a single authoritative nexus for reporting the effectiveness of different models on BEIR, which has led to difficulty in comparing different methods. To remedy this, we present an official self-service BEIR leaderboard that provides fair and consistent comparisons of retrieval models. By addressing both shortcomings, our work facilitates future explorations in a range of interesting research questions that BEIR enables.