Investigating the Successes and Failures of BERT for Passage Re-Ranking

TL;DR

This paper analyzes why fine-tuned BERT significantly improves passage re-ranking performance, focusing on the MS MARCO dataset, by empirically examining hypotheses and providing insights into its successes and failures.

Contribution

It offers a detailed empirical analysis of BERT's performance in passage re-ranking, identifying factors behind its success and limitations.

Findings

BERT achieves substantial improvements in passage re-ranking.

Certain hypotheses explain BERT's success in retrieval tasks.

Analysis reveals specific failure modes of BERT in re-ranking.

Abstract

The bidirectional encoder representations from transformers (BERT) model has recently advanced the state-of-the-art in passage re-ranking. In this paper, we analyze the results produced by a fine-tuned BERT model to better understand the reasons behind such substantial improvements. To this aim, we focus on the MS MARCO passage re-ranking dataset and provide potential reasons for the successes and failures of BERT for retrieval. In more detail, we empirically study a set of hypotheses and provide additional analysis to explain the successful performance of BERT.