Evaluating Cross-Lingual Unlearning in Multilingual Language Models

TL;DR

This paper evaluates cross-lingual unlearning in multilingual language models, revealing challenges in removing facts across languages and proposing subspace projection as an effective solution for multilingual forgetting.

Contribution

It provides the first comprehensive evaluation of cross-lingual unlearning and demonstrates the effectiveness of subspace-projection methods in multilingual models.

Findings

Most unlearning algorithms fail to remove facts outside the training language.

Subspace-projection outperforms other methods in cross-lingual forgetting.

Removing shared interlingua subspaces harms all languages.

Abstract

We present the first comprehensive evaluation of cross-lingual unlearning in multilingual LLMs. Using translated TOFU benchmarks in seven language/script variants, we test major unlearning algorithms and show that most fail to remove facts outside the training language, even when utility remains high. However, subspace-projection consistently outperforms the other methods, achieving strong cross-lingual forgetting with minimal degradation. Analysis of learned task subspaces reveals a shared interlingua structure: removing this shared subspace harms all languages, while removing language-specific components selectively affects one. These results demonstrate that multilingual forgetting depends on geometry in weight space, motivating subspace-based approaches for future unlearning systems.