Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearning

TL;DR

This paper presents STEU, a parameter-efficient method for class-level unlearning in clinical language models, which selectively updates token embeddings to forget specific information while preserving overall model utility.

Contribution

STEU introduces a novel, sparse embedding editing approach that unlearns targeted classes without retraining the entire model, maintaining performance with minimal parameter updates.

Findings

Achieves near-complete forgetting of target classes.

Maintains high task performance after minimal parameter modification.

Operates efficiently by updating only 0.19% of parameters.

Abstract

Machine unlearning is increasingly important for clinical language models, where privacy regulations and institutional policies may require removing sensitive information from deployed systems without retraining from scratch. In practice, deletion requests must balance effective forgetting of targeted information with preservation of model utility and minimal parameter modification. We introduce Sparse Token Embedding Unlearning (STEU), a parameter-efficient method for behavioral class-level unlearning that updates only PMI-selected token embeddings together with a small classifier head while keeping all encoder layers frozen. Across experiments on MIMIC-IV, MIMIC-III, and eICU using BioClinicalBERT, BERT-base, and DistilBERT, STEU consistently suppresses the target class while largely preserving retained task performance. In the primary MIMIC-IV setting, STEU achieves near-complete forgetting (forget F1 = 0.0004) while maintaining competitive retained utility (retain avg F1 = 0.4766) after modifying only 0.19\% of model parameters. These results suggest that targeted behavioral unlearning can be achieved through sparse embedding edits without modifying deeper encoder representations.