Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers

TL;DR

This paper introduces FLOPs-based metrics RPP and QPP for evaluating LLM rerankers, providing a hardware-independent way to measure efficiency and effectiveness trade-offs in information retrieval tasks.

Contribution

It proposes new FLOPs-based metrics and an estimator to better evaluate LLM rerankers' efficiency-effectiveness tradeoff independent of hardware and implementation details.

Findings

RPP and QPP offer consistent efficiency measurements across different hardware.

The FLOPs estimator accurately predicts model FLOPs without running experiments.

Comprehensive experiments reveal insights into the efficiency-effectiveness tradeoff.

Abstract

Large Language Models (LLMs) have recently been applied to reranking tasks in information retrieval, achieving strong performance. However, their high computational demands often hinder practical deployment. Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency, the number of forward passes, input tokens, and output tokens. However, these metrics depend on hardware and running-time choices (\eg parallel or not, batch size, etc), and often fail to account for model size, making it difficult to interpret and obscuring the evaluation of the efficiency-effectiveness tradeoff. To address this issue, we propose \ours\footnote{https://github.com/zhiyuanpeng/EER-FLOPs.} for LLM-based rerankers: RPP (ranking metrics per PetaFLOP), measuring how much ranking quality (e.g., NDCG or MRR) a method achieves per PetaFLOP, and QPP (queries per PetaFLOP), measuring how many queries can be processed per PetaFLOP. Accompanied by the new metrics, an interpretable FLOPs estimator is developed to estimate the FLOPs of an LLM-based reranker even without running any experiments. Based on the proposed metrics, we conduct comprehensive experiments to evaluate a wide range of LLM-based rerankers with different architectures, studying the efficiency-effectiveness trade-off and bringing this issue to the attention of the research community.