PaRaDe: Passage Ranking using Demonstrations with Large Language Models

TL;DR

This paper enhances large language model-based passage re-ranking by selecting effective few-shot demonstrations using a difficulty-based strategy, significantly improving relevance and also aiding question generation.

Contribution

It introduces a novel demonstration selection method based on difficulty, improving LLM re-ranking performance over traditional similarity-based approaches.

Findings

Adding even one demonstration improves re-ranking significantly

Difficulty-based demonstration selection outperforms semantic similarity methods

Demonstrations effective for both ranking and question generation

Abstract

Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.