Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness

TL;DR

This paper introduces IAM, a novel framework for measuring sample-level unlearning completeness in machine learning models, addressing limitations of existing methods by providing scalable, effective, and granular unlearning assessment.

Contribution

We propose IAM, a scalable and effective framework for quantifying unlearning completeness at the sample level, improving over existing binary and resource-intensive methods.

Findings

IAM performs well in binary inclusion tests for exact unlearning.

IAM shows high correlation with unlearning quality in approximate unlearning.

Applying IAM reveals risks of over- and under-unlearning in current algorithms.

Abstract

Growing concerns over data privacy and security highlight the importance of machine unlearning--removing specific data influences from trained models without full retraining. Techniques like Membership Inference Attacks (MIAs) are widely used to externally assess successful unlearning. However, existing methods face two key limitations: (1) maximizing MIA effectiveness (e.g., via online attacks) requires prohibitive computational resources, often exceeding retraining costs; (2) MIAs, designed for binary inclusion tests, struggle to capture granular changes in approximate unlearning. To address these challenges, we propose the Interpolated Approximate Measurement (IAM), a framework natively designed for unlearning inference. IAM quantifies sample-level unlearning completeness by interpolating the model's generalization-fitting behavior gap on queried samples. IAM achieves strong performance in binary inclusion tests for exact unlearning and high correlation for approximate unlearning--scalable to LLMs using just one pre-trained shadow model. We theoretically analyze how IAM's scoring mechanism maintains performance efficiently. We then apply IAM to recent approximate unlearning algorithms, revealing general risks of both over-unlearning and under-unlearning, underscoring the need for stronger safeguards in approximate unlearning systems. The code is available at https://github.com/Happy2Git/Unlearning_Inference_IAM.