On the Interpolation of Contextualized Term-based Ranking with BM25 for Query-by-Example Retrieval

TL;DR

This paper evaluates the effectiveness of contextualized term-based models TILDE and TILDEv2 in query-by-example retrieval, compares them with BM25, and demonstrates that combining these models improves retrieval performance.

Contribution

It introduces a score interpolation method combining TILDE (or TILDEv2) with BM25, showing significant improvements in query-by-example retrieval accuracy.

Findings

BM25 performs competitively with TILDE and TILDEv2 in QBE.

Interpolating scores from TILDE/TILDEv2 and BM25 improves retrieval results.

Contextualized models capture different relevance signals than BM25.

Abstract

Term-based ranking with pre-trained transformer-based language models has recently gained attention as they bring the contextualization power of transformer models into the highly efficient term-based retrieval. In this work, we examine the generalizability of two of these deep contextualized term-based models in the context of query-by-example (QBE) retrieval in which a seed document acts as the query to find relevant documents. In this setting -- where queries are much longer than common keyword queries -- BERT inference at query time is problematic as it involves quadratic complexity. We investigate TILDE and TILDEv2, both of which leverage BERT tokenizer as their query encoder. With this approach, there is no need for BERT inference at query time, and also the query can be of any length. Our extensive evaluation on the four QBE tasks of SciDocs benchmark shows that in a query-by-example retrieval setting TILDE and TILDEv2 are still less effective than a cross-encoder BERT ranker. However, we observe that BM25 could show a competitive ranking quality compared to TILDE and TILDEv2 which is in contrast to the findings about the relative performance of these three models on retrieval for short queries reported in prior work. This result raises the question about the use of contextualized term-based ranking models being beneficial in QBE setting. We follow-up on our findings by studying the score interpolation between the relevance score from TILDE (TILDEv2) and BM25. We conclude that these two contextualized term-based ranking models capture different relevance signals than BM25 and combining the different term-based rankers results in statistically significant improvements in QBE retrieval. Our work sheds light on the challenges of retrieval settings different from the common evaluation benchmarks.