A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADE

TL;DR

This study compares cross-encoder and LLM-based rerankers for SPLADE retrieval, revealing that traditional cross-encoders remain competitive, especially out-of-domain, despite GPT-4's impressive zero-shot performance, informing search system design.

Contribution

It provides a comprehensive evaluation of cross-encoders versus LLM rerankers across multiple datasets, highlighting their relative strengths and limitations in different scenarios.

Findings

Cross-encoders are hard to distinguish in in-domain re-ranking.

Model type and number of documents impact effectiveness out-of-domain.

GPT-4 shows impressive zero-shot performance but traditional models remain competitive.

Abstract

We present a comparative study between cross-encoder and LLMs rerankers in the context of re-ranking effective SPLADE retrievers. We conduct a large evaluation on TREC Deep Learning datasets and out-of-domain datasets such as BEIR and LoTTE. In the first set of experiments, we show how cross-encoder rerankers are hard to distinguish when it comes to re-rerank SPLADE on MS MARCO. Observations shift in the out-of-domain scenario, where both the type of model and the number of documents to re-rank have an impact on effectiveness. Then, we focus on listwise rerankers based on Large Language Models -- especially GPT-4. While GPT-4 demonstrates impressive (zero-shot) performance, we show that traditional cross-encoders remain very competitive. Overall, our findings aim to to provide a more nuanced perspective on the recent excitement surrounding LLM-based re-rankers -- by positioning them as another factor to consider in balancing effectiveness and efficiency in search systems.