Unlearning Evaluation through Subset Statistical Independence

TL;DR

This paper introduces a new subset-level evaluation framework for machine unlearning that uses statistical independence measures, specifically the Hilbert-Schmidt Independence Criterion, to assess unlearning effectiveness without retraining models.

Contribution

It proposes a novel, practical evaluation method based on statistical independence that does not require retraining or auxiliary classifiers, improving unlearning assessment.

Findings

Effectively distinguishes in-training from out-of-training data subsets.

Reliable differentiation of unlearning effectiveness.

Outperforms existing evaluation methods in various experiments.

Abstract

Evaluating machine unlearning remains challenging, as existing methods typically require retraining reference models or performing membership inference attacks, both of which rely on prior access to training configuration or supervision labels, making them impractical in realistic scenarios. Motivated by the fact that most unlearning algorithms remove a small, random subset of the training data, we propose a subset-level evaluation framework based on statistical independence. Specifically, we design a tailored use of the Hilbert-Schmidt Independence Criterion to assess whether the model outputs on a given subset exhibit statistical dependence, without requiring model retraining or auxiliary classifiers. Our method provides a simple, standalone evaluation procedure that aligns with unlearning workflows. Extensive experiments demonstrate that our approach reliably distinguishes in-training from out-of-training subsets and clearly differentiates unlearning effectiveness, even when existing evaluations fall short.