Two-stage Risk Control with Application to Ranked Retrieval

TL;DR

This paper introduces two-stage risk control methods tailored for ranked retrieval systems, leveraging sequential problem structure to provide theoretical guarantees and demonstrate effectiveness on large-scale datasets.

Contribution

The paper develops novel two-stage risk control methods based on LTT and CRC frameworks, specifically designed for ranked retrieval tasks, with reduced computational complexity and theoretical guarantees.

Findings

Effective risk control demonstrated on large-scale datasets

Theoretical guarantees established for proposed methods

Tailored loss functions improve ranking performance

Abstract

Practical machine learning systems often operate in multiple sequential stages, as seen in ranking and recommendation systems, which typically include a retrieval phase followed by a ranking phase. Effectively assessing prediction uncertainty and ensuring effective risk control in such systems pose significant challenges due to their inherent complexity. To address these challenges, we developed two-stage risk control methods based on the recently proposed learn-then-test (LTT) and conformal risk control (CRC) frameworks. Unlike the methods in prior work that address multiple risks, our approach leverages the sequential nature of the problem, resulting in reduced computational burden. We provide theoretical guarantees for our proposed methods and design novel loss functions tailored for ranked retrieval tasks. The effectiveness of our approach is validated through experiments on two large-scale, widely-used datasets: MSLR-Web and Yahoo LTRC.