SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing

TL;DR

This paper introduces SMI, a novel, training-free framework for auditing machine unlearning that accurately estimates the extent of forgotten data without relying on membership inference attacks.

Contribution

SMI provides a theoretically grounded, efficient alternative to MIAs for unlearned model auditing, eliminating shadow model training and improving reliability.

Findings

SMI outperforms MIA-based baselines in experiments.

Failed membership inference does not guarantee data has been forgotten.

SMI offers bootstrap ranges for auditing reliability.

Abstract

Machine unlearning (MU) is essential for enforcing the right to be forgotten in machine learning systems. A key challenge of MU is how to reliably audit whether a model has truly forgotten specified training data. Membership Inference Attacks (MIAs) are widely used for unlearned model auditing, where samples that evade membership detection are regarded as successfully forgotten. We show this assumption is fundamentally flawed: failed membership inference does not imply true forgetting. We prove that unlearned samples occupy fundamentally different positions in the feature space than non-member samples, making this alignment bias unavoidable and unobservable, which leads to systematically optimistic evaluations of unlearning performance. Meanwhile, training shadow models for MIA incurs substantial computational overhead. To address both limitations, we propose Statistical Membership Inference (SMI), a training-free auditing framework that reformulates auditing as estimating the non-member mixture proportion in the unlearned feature distribution. Beyond estimating the forgetting rate, SMI also provides bootstrap reference ranges for quantified auditing reliability. Extensive experiments show that SMI consistently outperforms all MIA-based baselines, with no shadow model training required. Overall, SMI establishes a principled and efficient alternative to MIA-based auditing methods, with both theoretical guarantees and strong empirical performance.