Semi-Supervised Risk Control via Prediction-Powered Inference

TL;DR

This paper introduces a semi-supervised calibration method for risk-controlling prediction sets that uses unlabeled data to improve error rate tuning, demonstrated through real-data experiments.

Contribution

It proposes a novel semi-supervised calibration procedure for RCPS that leverages unlabeled data to reduce conservativeness and improve error control.

Findings

Improved error rate control in limited data scenarios

Effective application to few-shot image classification

Enhanced early time series classification performance

Abstract

The risk-controlling prediction sets (RCPS) framework is a general tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. The key idea behind this framework is to use labeled hold-out calibration data to tune a hyper-parameter that affects the error rate of the resulting prediction rule. However, the limitation of such a calibration scheme is that with limited hold-out data, the tuned hyper-parameter becomes noisy and leads to a prediction rule with an error rate that is often unnecessarily conservative. To overcome this sample-size barrier, we introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper-parameter without compromising statistical validity. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks. We demonstrate the benefits and validity of our proposal through two real-data experiments: few-shot image classification and early time series classification.