Streamlining Conformal Information Retrieval via Score Refinement

TL;DR

This paper proposes a score refinement technique for conformal information retrieval that reduces set size and maintains guarantees, improving efficiency and response times in IR applications.

Contribution

The paper introduces a simple monotone score transformation method that produces smaller conformal sets without losing statistical guarantees.

Findings

Smaller conformal sets achieved across multiple benchmarks

Maintains statistical guarantees of relevance inclusion

Improves computational efficiency and response times

Abstract

Information retrieval (IR) methods, like retrieval augmented generation, are fundamental to modern applications but often lack statistical guarantees. Conformal prediction addresses this by retrieving sets guaranteed to include relevant information, yet existing approaches produce large-sized sets, incurring high computational costs and slow response times. In this work, we introduce a score refinement method that applies a simple monotone transformation to retrieval scores, leading to significantly smaller conformal sets while maintaining their statistical guarantees. Experiments on various BEIR benchmarks validate the effectiveness of our approach in producing compact sets containing relevant information.