Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

TL;DR

This paper introduces new metrics, KSS and KPS, to evaluate how effectively multilingual machine unlearning removes cross-linguistic information, addressing limitations of prior evaluations.

Contribution

It proposes two novel evaluation metrics for multilingual machine unlearning that better capture cross-linguistic information removal and provides comprehensive analysis using these metrics.

Findings

KSS and KPS effectively measure unlearning quality across languages.

Evaluation reveals phenomena unique to multilingual unlearning.

Insights inform better unlearning methods for multilingual models.

Abstract

While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII). For LLMs trained on multilingual corpora, Multilingual Machine Unlearning (MMU) aims to remove information across multiple languages. However, prior MMU evaluations fail to capture such cross-linguistic distribution of information, being largely limited to direct extensions of per-language evaluation protocols. To this end, we propose two metrics to evaluate the information spread across languages: the Knowledge Separability Score (KSS) and the Knowledge Persistence Score (KPS). KSS measures the overall unlearning quality across multiple languages, while KPS more specifically aims to assess consistent removal of information among different language pairs. We evaluated various unlearning methods in the multilingual setting with these metrics and conducted comprehensive analyses. Through our investigation, we provide insights into unique phenomena exclusive to MMU and offer a new perspective on MMU evaluation.