Risk Controlled Image Retrieval

TL;DR

This paper introduces Risk Controlled Image Retrieval (RCIR), a novel method that guarantees with high probability that retrieved image sets contain true nearest neighbors, addressing reliability concerns in image retrieval.

Contribution

RCIR is the first approach to provide coverage guarantees in image retrieval, integrating uncertainty quantification with probabilistic guarantees for retrieval sets.

Findings

RCIR achieves coverage guarantees on four real-world datasets.

The method is compatible with existing uncertainty-aware retrieval systems.

RCIR improves reliability without sacrificing retrieval performance.

Abstract

Most image retrieval research prioritizes improving predictive performance, often overlooking situations where the reliability of predictions is equally important. The gap between model performance and reliability requirements highlights the need for a systematic approach to analyze and address the risks associated with image retrieval. Uncertainty quantification technique can be applied to mitigate this issue by assessing uncertainty for retrieval sets, but it provides only a heuristic estimate of uncertainty rather than a guarantee. To address these limitations, we present Risk Controlled Image Retrieval (RCIR), which generates retrieval sets with coverage guarantee, i.e., retrieval sets that are guaranteed to contain the true nearest neighbors with a predefined probability. RCIR can be easily integrated with existing uncertainty-aware image retrieval systems, agnostic to data distribution and model selection. To the best of our knowledge, this is the first work that provides coverage guarantees to image retrieval. The validity and efficiency of RCIR are demonstrated on four real-world datasets: CAR-196, CUB-200, Pittsburgh, and ChestX-Det.