Fast Online Value-Maximizing Prediction Sets with Conformal Cost Control

TL;DR

FavMac is a versatile online method that maximizes the value of multi-label predictions while strictly controlling costs, suitable for large-scale real-world applications like healthcare, with theoretical guarantees and demonstrated superior performance.

Contribution

Introduces FavMac, a general pipeline for value maximization with cost control in multi-label prediction, featuring online updates and distribution-free guarantees.

Findings

FavMac achieves higher value than baselines.

FavMac maintains strict cost control.

Effective in healthcare and synthetic datasets.

Abstract

Many real-world multi-label prediction problems involve set-valued predictions that must satisfy specific requirements dictated by downstream usage. We focus on a typical scenario where such requirements, separately encoding $\textit{value}$ and $\textit{cost}$, compete with each other. For instance, a hospital might expect a smart diagnosis system to capture as many severe, often co-morbid, diseases as possible (the value), while maintaining strict control over incorrect predictions (the cost). We present a general pipeline, dubbed as FavMac, to maximize the value while controlling the cost in such scenarios. FavMac can be combined with almost any multi-label classifier, affording distribution-free theoretical guarantees on cost control. Moreover, unlike prior works, it can handle real-world large-scale applications via a carefully designed online update mechanism, which is of independent interest. Our methodological and theoretical contributions are supported by experiments on several healthcare tasks and synthetic datasets - FavMac furnishes higher value compared with several variants and baselines while maintaining strict cost control. Our code is available at https://github.com/zlin7/FavMac