FAME: Fictional Actors for Multilingual Erasure

TL;DR

FAME is a multilingual synthetic benchmark designed to evaluate machine unlearning techniques in large language models, enabling systematic, controlled testing of forgetting specific information across five languages.

Contribution

Introduces FAME, a novel multilingual synthetic benchmark with structured data for evaluating entity- and instance-level unlearning in LLMs across five languages.

Findings

Supports entity-level and instance-level unlearning scenarios

Enables systematic comparison of unlearning techniques across languages

Uses fictional data to ensure controlled evaluation

Abstract

LLMs trained on web-scale data raise concerns about privacy and the right to be forgotten. To address these issues, Machine Unlearning provides techniques to remove specific information from trained models without retraining from scratch. However, existing benchmarks for evaluating unlearning in LLMs face two major limitations: they focus only on English and support only entity-level forgetting (removing all information about a person). We introduce FAME (Fictional Actors for Multilingual Erasure), a synthetic benchmark for evaluating Machine Unlearning across five languages: English, French, German, Italian, and Spanish. FAME contains 1,000 fictional actor biographies and 20,000 question-answer pairs. Each biography includes information on 20 topics organized into structured categories (biography, career, achievements, personal information). This design enables both entity-level unlearning (i.e., forgetting entire identities) and instance-level unlearning (i.e., forgetting specific facts while retaining others). We provide two dataset splits to support these two different unlearning scenarios and enable systematic comparison of unlearning techniques across languages. Since FAME uses entirely fictional data, it ensures that the information was never encountered during model pretraining, allowing for a controlled evaluation of unlearning methods.