PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models

TL;DR

PyTerrier-GenRank is a plugin that simplifies the process of using large language models for reranking in information retrieval tasks, supporting various prompting strategies and validated with major LLM providers.

Contribution

It introduces a PyTerrier plugin that streamlines LLM-based reranking experiments, handling hyperparameter tuning and supporting multiple ranking strategies.

Findings

Supports popular ranking strategies like pointwise and listwise prompting.

Validated with HuggingFace and OpenAI endpoints.

Facilitates hyperparameter experimentation for LLM rerankers.

Abstract

Using LLMs as rerankers requires experimenting with various hyperparameters, such as prompt formats, model choice, and reformulation strategies. We introduce PyTerrier-GenRank, a PyTerrier plugin to facilitate seamless reranking experiments with LLMs, supporting popular ranking strategies like pointwise and listwise prompting. We validate our plugin through HuggingFace and OpenAI hosted endpoints.