Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective

TL;DR

This paper introduces RULI, a new evaluation framework for inexact machine unlearning that exposes privacy vulnerabilities and assesses unlearning efficacy at a granular level, revealing significant weaknesses in current methods.

Contribution

The paper proposes RULI, a dual-objective attack framework that improves evaluation of unlearning methods by addressing key gaps and providing fine-grained privacy and efficacy assessments.

Findings

RULI achieves higher attack success rates than existing evaluations.

State-of-the-art unlearning methods have significant privacy vulnerabilities.

RULI is validated on image and text datasets across various tasks.

Abstract

Machine unlearning focuses on efficiently removing specific data from trained models, addressing privacy and compliance concerns with reasonable costs. Although exact unlearning ensures complete data removal equivalent to retraining, it is impractical for large-scale models, leading to growing interest in inexact unlearning methods. However, the lack of formal guarantees in these methods necessitates the need for robust evaluation frameworks to assess their privacy and effectiveness. In this work, we first identify several key pitfalls of the existing unlearning evaluation frameworks, e.g., focusing on average-case evaluation or targeting random samples for evaluation, incomplete comparisons with the retraining baseline. Then, we propose RULI (Rectified Unlearning Evaluation Framework via Likelihood Inference), a novel framework to address critical gaps in the evaluation of inexact unlearning methods. RULI introduces a dual-objective attack to measure both unlearning efficacy and privacy risks at a per-sample granularity. Our findings reveal significant vulnerabilities in state-of-the-art unlearning methods, where RULI achieves higher attack success rates, exposing privacy risks underestimated by existing methods. Built on a game-based foundation and validated through empirical evaluations on both image and text data (spanning tasks from classification to generation), RULI provides a rigorous, scalable, and fine-grained methodology for evaluating unlearning techniques.