Conformal Unlearning: A New Paradigm for Unlearning in Conformal Predictors

TL;DR

This paper introduces a new approach to conformal unlearning that guarantees effective removal of specific data while maintaining predictive validity, addressing limitations of existing methods that lack uncertainty-aware evaluation.

Contribution

It formalizes conformal unlearning with finite-sample guarantees, proposes practical metrics, and develops an algorithm that effectively removes targeted data without retraining from scratch.

Findings

Effectively removes targeted data while preserving utility.

Provides finite-sample, uncertainty-aware guarantees.

Outperforms existing methods on vision and text benchmarks.

Abstract

Conformal unlearning aims to ensure that a trained conformal predictor miscovers data points with specific shared characteristics, such as those from a particular label class, associated with a specific user, or belonging to a defined cluster, while maintaining valid coverage on the remaining data. Existing machine unlearning methods, which typically approximate a model retrained from scratch after removing the data to be forgotten, face significant challenges when applied to conformal unlearning. These methods often lack rigorous, uncertainty-aware statistical measures to evaluate unlearning effectiveness and exhibit a mismatch between their degraded performance on forgotten data and the frequency with which that data are still correctly covered by conformal predictors-a phenomenon we term ''fake conformal unlearning''. To address these limitations, we propose a new paradigm for conformal machine unlearning that provides finite-sample, uncertainty-aware guarantees on unlearning performance without relying on a retrained model as a reference. We formalize conformal unlearning to require high coverage on retained data and high miscoverage on forgotten data, introduce practical empirical metrics for evaluation, and present an algorithm that optimizes these conformal objectives. Extensive experiments on vision and text benchmarks demonstrate that the proposed approach effectively removes targeted information while preserving utility.