Stake the Points: Structure-Faithful Instance Unlearning

TL;DR

This paper introduces a structure-faithful unlearning framework using semantic anchors to preserve knowledge organization, improving privacy and utility balance in models across various tasks.

Contribution

It proposes a novel framework with semantic anchors to maintain knowledge structure during unlearning, addressing a key limitation of existing methods.

Findings

Achieves 32.9% performance gain in image classification

Improves retrieval and face recognition results by over 19%

Enhances the deletion-retention balance and generalization

Abstract

Machine unlearning (MU) addresses privacy risks in pretrained models. The main goal of MU is to remove the influence of designated data while preserving the utility of retained knowledge. Achieving this goal requires preserving semantic relations among retained instances, which existing studies often overlook. We observe that without such preservation, models suffer from progressive structural collapse, undermining both the deletion-retention balance. In this work, we propose a novel structure-faithful framework that introduces stakes, i.e., semantic anchors that serve as reference points to maintain the knowledge structure. By leveraging these anchors, our framework captures and stabilizes the semantic organization of knowledge. Specifically, we instantiate the anchors from language-driven attribute descriptions encoded by a semantic encoder (e.g., CLIP). We enforce preservation of the knowledge structure via structure-aware alignment and regularization: the former aligns the organization of retained knowledge before and after unlearning around anchors, while the latter regulates updates to structure-critical parameters. Results from image classification, retrieval, and face recognition show average gains of 32.9%, 22.5%, and 19.3% in performance, balancing the deletion-retention trade-off and enhancing generalization.