Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models

TL;DR

This paper proposes an iterative unlearning framework for generative language models that effectively removes sensitive information while preserving model performance, addressing privacy concerns and data unavailability issues.

Contribution

The paper introduces the ICU framework with three modules for targeted unlearning, preserving model capabilities, and iterative refinement, advancing privacy-preserving NLP techniques.

Findings

Effective removal of sensitive data demonstrated

Maintains overall model performance

Applicable without access to original training data

Abstract

Recent advances in machine learning, particularly in Natural Language Processing (NLP), have produced powerful models trained on vast datasets. However, these models risk leaking sensitive information, raising privacy concerns. In response, regulatory measures such as the European Union's General Data Protection Regulation (GDPR) have driven increasing interest in Machine Unlearning techniques, which enable models to selectively forget specific data entries. Early unlearning approaches primarily relied on pre-processing methods, while more recent research has shifted towards training-based solutions. Despite their effectiveness, a key limitation persists: most methods require access to original training data, which is often unavailable. Additionally, directly applying unlearning techniques bears the cost of undermining the model's expressive capabilities. To address these challenges, we introduce the Iterative Contrastive Unlearning (ICU) framework, which consists of three core components: A Knowledge Unlearning Induction module designed to target specific knowledge for removal using an unlearning loss; A Contrastive Learning Enhancement module to preserve the model's expressive capabilities against the pure unlearning goal; And an Iterative Unlearning Refinement module that dynamically adjusts the unlearning process through ongoing evaluation and updates. Experimental results demonstrate the efficacy of our ICU method in unlearning sensitive information while maintaining the model's overall performance, offering a promising solution for privacy-conscious machine learning applications.