RankLLM: A Python Package for Reranking with LLMs

TL;DR

RankLLM is an open-source Python package designed for flexible, configurable reranking of documents using large language models, supporting multiple models and evaluation methods to facilitate research and practical applications.

Contribution

It introduces a modular, highly configurable reranking package supporting various LLMs, with integrated evaluation and analysis tools for improved usability and reliability.

Findings

Reproduced results from recent LLM reranking models.

Supported a wide range of proprietary and open-source LLMs.

Enhanced usability with integrated evaluation and analysis modules.

Abstract

The adoption of large language models (LLMs) as rerankers in multi-stage retrieval systems has gained significant traction in academia and industry. These models refine a candidate list of retrieved documents, often through carefully designed prompts, and are typically used in applications built on retrieval-augmented generation (RAG). This paper introduces RankLLM, an open-source Python package for reranking that is modular, highly configurable, and supports both proprietary and open-source LLMs in customized reranking workflows. To improve usability, RankLLM features optional integration with Pyserini for retrieval and provides integrated evaluation for multi-stage pipelines. Additionally, RankLLM includes a module for detailed analysis of input prompts and LLM responses, addressing reliability concerns with LLM APIs and non-deterministic behavior in Mixture-of-Experts (MoE) models. This paper presents the architecture of RankLLM, along with a detailed step-by-step guide and sample code. We reproduce results from RankGPT, LRL, RankVicuna, RankZephyr, and other recent models. RankLLM integrates with common inference frameworks and a wide range of LLMs. This compatibility allows for quick reproduction of reported results, helping to speed up both research and real-world applications. The complete repository is available at rankllm.ai, and the package can be installed via PyPI.