How Different are Pre-trained Transformers for Text Ranking?

TL;DR

This paper compares pre-trained transformer models to traditional ranking methods in passage retrieval, revealing significant differences in relevance notions and highlighting areas for future research.

Contribution

It provides a detailed analysis of how BERT-based neural rankers differ from traditional methods, especially in the context of MS Marco datasets.

Findings

Neural rankers often prioritize different relevance signals than traditional models.

BERT models can retrieve documents missed by traditional systems, enhancing recall.

Substantial differences in relevance understanding suggest avenues for improving neural ranking models.

Abstract

In recent years, large pre-trained transformers have led to substantial gains in performance over traditional retrieval models and feedback approaches. However, these results are primarily based on the MS Marco/TREC Deep Learning Track setup, with its very particular setup, and our understanding of why and how these models work better is fragmented at best. We analyze effective BERT-based cross-encoders versus traditional BM25 ranking for the passage retrieval task where the largest gains have been observed, and investigate two main questions. On the one hand, what is similar? To what extent does the neural ranker already encompass the capacity of traditional rankers? Is the gain in performance due to a better ranking of the same documents (prioritizing precision)? On the other hand, what is different? Can it retrieve effectively documents missed by traditional systems (prioritizing recall)? We discover substantial differences in the notion of relevance identifying strengths and weaknesses of BERT that may inspire research for future improvement. Our results contribute to our understanding of (black-box) neural rankers relative to (well-understood) traditional rankers, help understand the particular experimental setting of MS-Marco-based test collections.