A Linear Expectation Constraint for Selective Prediction and Routing with False-Discovery Control

TL;DR

This paper introduces LEC, a new framework for selective prediction that guarantees false discovery rate control using a linear expectation constraint, improving reliability and sample retention in AI systems.

Contribution

LEC reformulates selective prediction as a decision problem with a linear expectation constraint, enabling FDR control with only calibration data and extending to multi-model routing.

Findings

LEC achieves tighter FDR control than previous methods.

LEC significantly increases sample retention in QA and VQA tasks.

Experiments demonstrate improved reliability and efficiency of the proposed approach.

Abstract

Foundation models often generate unreliable answers, while heuristic uncertainty estimators fail to fully distinguish correct from incorrect outputs, causing users to accept erroneous answers without statistical guarantees. We address this through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level. To this end, we propose LEC, a principled framework that reframes selective prediction as a decision problem governed by a linear expectation constraint over selection and error indicators. Under this formulation, we derive a finite-sample sufficient condition that relies only on a held-out set of exchangeable calibration data, enabling the computation of an FDR-constrained, retention-maximizing threshold. Furthermore, we extend LEC to two-model routing systems: if the primary model's uncertainty exceeds its calibrated threshold, the input is delegated to a subsequent model, while maintaining system-level FDR control. Experiments on both closed-ended and open-ended question answering (QA) and vision question answering (VQA) demonstrate that LEC achieves tighter FDR control and substantially improves sample retention compared to prior approaches.