Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning

TL;DR

This paper introduces reliable conformal prediction sets that maintain coverage guarantees even under data poisoning attacks by using aggregation methods on predictions from multiple data partitions.

Contribution

It proposes the first efficient method for conformal prediction with provable reliability under poisoning, using smoothed scores and multiple calibration sets.

Findings

Achieves strong reliability under poisoning attacks

Maintains coverage and utility on clean data

Validates approach on image classification tasks

Abstract

Conformal prediction provides model-agnostic and distribution-free uncertainty quantification through prediction sets that are guaranteed to include the ground truth with any user-specified probability. Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data, which can significantly alter prediction sets in practice. As a solution, we propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning. To ensure reliability under training poisoning, we introduce smoothed score functions that reliably aggregate predictions of classifiers trained on distinct partitions of the training data. To ensure reliability under calibration poisoning, we construct multiple prediction sets, each calibrated on distinct subsets of the calibration data. We then aggregate them into a majority prediction set, which includes a class only if it appears in a majority of the individual sets. Both proposed aggregations mitigate the influence of datapoints in the training and calibration data on the final prediction set. We experimentally validate our approach on image classification tasks, achieving strong reliability while maintaining utility and preserving coverage on clean data. Overall, our approach represents an important step towards more trustworthy uncertainty quantification in the presence of data poisoning.