Delete and Retain: Efficient Unlearning for Document Classification

TL;DR

This paper introduces Hessian Reassignment, a fast, model-agnostic method for class-level unlearning in document classifiers, achieving near full retrain accuracy and improved privacy guarantees.

Contribution

The paper presents a novel two-step unlearning approach that efficiently removes class influence from document classifiers, with a decision-space guarantee and improved privacy.

Findings

Achieves accuracy close to full retraining after unlearning

Runs orders of magnitude faster than full retraining

Reduces membership-inference advantage on removed class

Abstract

Machine unlearning aims to efficiently remove the influence of specific training data from a model without full retraining. While much progress has been made in unlearning for LLMs, document classification models remain relatively understudied. In this paper, we study class-level unlearning for document classifiers and present Hessian Reassignment, a two-step, model-agnostic solution. First, we perform a single influence-style update that subtracts the contribution of all training points from the target class by solving a Hessian-vector system with conjugate gradients, requiring only gradient and Hessian-vector products. Second, in contrast to common unlearning baselines that randomly reclassify deleted-class samples, we enforce a decision-space guarantee via Top-1 classification. On standard text benchmarks, Hessian Reassignment achieves retained-class accuracy close to full retrain-without-class while running orders of magnitude faster. Additionally, it consistently lowers membership-inference advantage on the removed class, measured with pooled multi-shadow attacks. These results demonstrate a practical, principled path to efficient class unlearning in document classification.