Suppression or Deletion: A Restoration-Based Representation-Level Analysis of Machine Unlearning

TL;DR

This paper introduces a restoration-based framework using autoencoders to evaluate whether machine unlearning methods truly delete sensitive information at the representation level, revealing that many methods only suppress rather than delete data.

Contribution

The authors propose a novel restoration-based analysis framework to assess the effectiveness of unlearning methods at the representation level, highlighting limitations of existing output-based metrics.

Findings

Most unlearning methods only suppress information, not delete it.

Semantic features from pretraining are retained even after retraining.

Representation-level retention poses privacy risks not detected by output metrics.

Abstract

As pretrained models are increasingly shared on the web, ensuring that models can forget or delete sensitive, copyrighted, or private information upon request has become crucial. Machine unlearning has been proposed to address this challenge. However, current evaluations for unlearning methods rely on output-based metrics, which cannot verify whether information is completely deleted or merely suppressed at the representation level, where suppression is insufficient for true unlearning. To address this gap, we propose a novel restoration-based analysis framework that uses Sparse Autoencoders to identify class-specific expert features in intermediate layers and applies inference-time steering to quantitatively distinguish between suppression and deletion. Applying our framework to 12 major unlearning methods in image classification tasks, we find that most methods achieve high restoration rates of unlearned information, indicating that they only suppress information at the decision-boundary level, while preserving semantic features in intermediate representations. Notably, even retraining from pretrained checkpoints shows high restoration, revealing that robust semantic features inherited from pretraining are not removed by retraining. These results demonstrate that representation-level retention poses significant risks overlooked by output-based metrics, highlighting the need for new unlearning evaluation criteria. We propose new evaluation guidelines that prioritize representation-level verification, especially for privacy-critical applications in the era of pre-trained models.