Erase at the Core: Representation Unlearning for Machine Unlearning

TL;DR

This paper introduces the Erase at the Core framework, which enforces comprehensive forgetting across all network layers, not just the final output, thereby improving the effectiveness of machine unlearning at both logit and representation levels.

Contribution

The paper proposes a novel multi-layer contrastive unlearning framework that enhances representation-level forgetting and can be integrated into existing unlearning methods.

Findings

EC achieves effective logit-level forgetting.

EC substantially reduces representational similarity to the original model.

EC improves representation-level forgetting while maintaining performance on the retain set.

Abstract

Many approximate machine unlearning methods demonstrate strong logit-level forgetting -- such as near-zero accuracy on the forget set -- yet continue to preserve substantial information within their internal feature representations. We refer to this discrepancy as superficial forgetting. Recent studies indicate that most existing unlearning approaches primarily alter the final classifier, leaving intermediate representations largely unchanged and highly similar to those of the original model. To address this limitation, we introduce the Erase at the Core (EC), a framework designed to enforce forgetting throughout the entire network hierarchy. EC integrates multi-layer contrastive unlearning on the forget set with retain set preservation through deeply supervised learning. Concretely, EC attaches auxiliary modules to intermediate layers and applies both contrastive unlearning and cross-entropy losses at each supervision point, with layer-wise weighted losses. Experimental results show that EC not only achieves effective logit-level forgetting, but also substantially reduces representational similarity to the original model across intermediate layers. Furthermore, EC is model-agnostic and can be incorporated as a plug-in module into existing unlearning methods, improving representation-level forgetting while maintaining performance on the retain set.