Transductive Conformal Inference for Full Ranking

TL;DR

This paper presents a conformal prediction-based method to quantify uncertainty in full ranking tasks, especially when only partial ground truth is known, providing valid prediction sets and error bounds for new items.

Contribution

It introduces a novel conformal inference approach tailored for full ranking scenarios with partial ground truth, addressing calibration challenges and controlling false coverage.

Findings

Effective in synthetic and real data experiments

Compatible with state-of-the-art ranking algorithms

Provides valid uncertainty quantification and error bounds

Abstract

We introduce a method based on Conformal Prediction (CP) to quantify the uncertainty of full ranking algorithms. We focus on a specific scenario where $n+m$ items are to be ranked by some ``black box'' algorithm. It is assumed that the relative (ground truth) ranking of $n$ of them is known. The objective is then to quantify the error made by the algorithm on the ranks of the $m$ new items among the total $(n+m)$. In such a setting, the true ranks of the $n$ original items in the total $(n+m)$ depend on the (unknown) true ranks of the $m$ new ones. Consequently, we have no direct access to a calibration set to apply a classical CP method. To address this challenge, we propose to construct distribution-free bounds of the unknown conformity scores using recent results on the distribution of conformal p-values. Using these scores upper bounds, we provide valid prediction sets for the rank of any item. We also control the false coverage proportion, a crucial quantity when dealing with multiple prediction sets. Finally, we empirically show on both synthetic and real data the efficiency of our CP method for state-of-the-art algorithms such as RankNet or LambdaMart.