MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

TL;DR

This paper analyzes the MS MARCO benchmark to understand how evaluation design influences research outcomes, emphasizing the importance of developing robust, generalizable ranking models in large-data settings.

Contribution

It offers a critical analysis of evaluation practices in large-scale ranking benchmarks and proposes best practices for fostering robust, widely applicable models.

Findings

Evaluation design impacts research outcomes and model robustness.

Current benchmarks may encourage overfitting to specific metrics.

Recommendations for improving evaluation validity and usefulness.

Abstract

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.