Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models

TL;DR

This study compares how shallow and deep relevance judgments influence BERT-based reranking models in neural IR, finding that shallow judgments improve generalization, while deep judgments may benefit from more negative examples.

Contribution

It provides an empirical comparison of shallow versus deep relevance judgments and their effects on BERT reranking performance in neural IR.

Findings

Shallow-judged datasets enhance model generalization.

Deep-judged datasets can be improved with more negative examples.

Results are based on MS MARCO and LongEval collections.

Abstract

This paper investigates the impact of shallow versus deep relevance judgments on the performance of BERT-based reranking models in neural Information Retrieval. Shallow-judged datasets, characterized by numerous queries each with few relevance judgments, and deep-judged datasets, involving fewer queries with extensive relevance judgments, are compared. The research assesses how these datasets affect the performance of BERT-based reranking models trained on them. The experiments are run on the MS MARCO and LongEval collections. Results indicate that shallow-judged datasets generally enhance generalization and effectiveness of reranking models due to a broader range of available contexts. The disadvantage of the deep-judged datasets might be mitigated by a larger number of negative training examples.