Towards Reliable Testing of Machine Unlearning

TL;DR

This paper proposes a causal, pathway-centric testing framework for machine unlearning to ensure models no longer rely on deleted data, addressing practical deployment challenges.

Contribution

It introduces a causal fuzzing approach for comprehensive, debuggable, and cost-effective unlearning testing applicable to black-box models.

Findings

Standard attribution checks can miss residual influence.

Causal testing uncovers proxy and subgroup effects.

Proof-of-concept demonstrates effectiveness of the approach.

Abstract

Machine learning components are now central to AI-infused software systems, from recommendations and code assistants to clinical decision support. As regulations and governance frameworks increasingly require deleting sensitive data from deployed models, machine unlearning is emerging as a practical alternative to full retraining. However, unlearning introduces a software quality-assurance challenge: under realistic deployment constraints and imperfect oracles, how can we test that a model no longer relies on targeted information? This paper frames unlearning testing as a first-class software engineering problem. We argue that practical unlearning tests must provide (i) thorough coverage over proxy and mediated influence pathways, (ii) debuggable diagnostics that localize where leakage persists, (iii) cost-effective regression-style execution under query budgets, and (iv) black-box applicability for API-deployed models. We outline a causal, pathway-centric perspective, causal fuzzing, that generates budgeted interventions to estimate residual direct and indirect effects and produce actionable "leakage reports". Proof-of-concept results illustrate that standard attribution checks can miss residual influence due to proxy pathways, cancellation effects, and subgroup masking, motivating causal testing as a promising direction for unlearning testing.