Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis

TL;DR

This paper introduces the task of entity-level unlearning in large language models, systematically evaluates existing algorithms, and identifies key factors affecting unlearning effectiveness, highlighting current challenges and vulnerabilities.

Contribution

It defines entity-level unlearning as a new task, evaluates trending algorithms, and analyzes factors influencing unlearning performance in LLMs.

Findings

Current methods struggle with effective entity-level unlearning.

Knowledge coverage and forget set size significantly impact unlearning success.

Entities from fine-tuning are more vulnerable to unlearning.

Abstract

Large language model unlearning has garnered increasing attention due to its potential to address security and privacy concerns, leading to extensive research in the field. However, much of this research has concentrated on instance-level unlearning, specifically targeting the removal of predefined instances containing sensitive content. This focus has left a significant gap in the exploration of full entity-level unlearning, which is critical in real-world scenarios such as copyright protection. To this end, we propose a novel task of Entity-level unlearning, which aims to erase entity-related knowledge from the target model completely. To thoroughly investigate this task, we systematically evaluate trending unlearning algorithms, revealing that current methods struggle to achieve effective entity-level unlearning. Then, we further explore the factors that influence the performance of the unlearning algorithms, identifying that knowledge coverage and the size of the forget set play pivotal roles. Notably, our analysis also uncovers that entities introduced through fine-tuning are more vulnerable to unlearning than pre-trained entities. These findings collectively offer valuable insights for advancing entity-level unlearning for LLMs.