LUME: LLM Unlearning with Multitask Evaluations

TL;DR

This paper introduces LUME, a multi-task benchmark for evaluating unlearning methods in large language models, focusing on removing sensitive, copyrighted, or private content without full retraining.

Contribution

It presents a new multi-task benchmark (LUME), releases fine-tuned LLMs, and evaluates recent unlearning algorithms with detailed metrics and analysis.

Findings

Unlearning algorithms show varied effectiveness across tasks.

Some methods struggle with sensitive and copyrighted content.

Benchmark reveals limitations of current unlearning techniques.

Abstract

Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.