Repeatability Corner Cases in Document Ranking: The Impact of Score Ties
Jimmy Lin, Peilin Yang

TL;DR
This paper investigates how score ties caused by multi-threaded indexing in Lucene affect the repeatability of document ranking experiments, highlighting the variability introduced and proposing external document ids as a solution.
Contribution
It quantifies the impact of score ties on ranking variability and discusses the trade-offs of using external document ids for tie-breaking.
Findings
Score ties cause significant variability in ranking results.
Using external document ids improves repeatability but incurs efficiency costs.
Multi-threaded indexing complicates consistent document id assignment.
Abstract
Document ranking experiments should be repeatable. However, the interaction between multi-threaded indexing and score ties during retrieval may yield non-deterministic rankings, making repeatability not as trivial as one might imagine. In the context of the open-source Lucene search engine, score ties are broken by internal document ids, which are assigned at index time. Due to multi-threaded indexing, which makes experimentation with large modern document collections practical, internal document ids are not assigned consistently between different index instances of the same collection, and thus score ties are broken unpredictably. This short paper examines the effectiveness impact of such score ties, quantifying the variability that can be attributed to this phenomenon. The obvious solution to this non-determinism and to ensure repeatable document ranking is to break score ties using…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInformation Retrieval and Search Behavior · Advanced Database Systems and Queries · Data Management and Algorithms
