Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models

TL;DR

This paper introduces a novel method for selectively unlearning sensitive information across multiple languages in multilingual language models, addressing privacy concerns while preserving model performance.

Contribution

It proposes an adaptive unlearning scheme that effectively erases knowledge in specific languages without degrading overall multilingual model performance.

Findings

Effective unlearning across languages demonstrated

Outperforms existing unlearning baselines

Maintains model performance after unlearning

Abstract

Pretrained language models memorize vast amounts of information, including private and copyrighted data, raising significant safety concerns. Retraining these models after excluding sensitive data is prohibitively expensive, making machine unlearning a viable, cost-effective alternative. Previous research has focused on machine unlearning for monolingual models, but we find that unlearning in one language does not necessarily transfer to others. This vulnerability makes models susceptible to low-resource language attacks, where sensitive information remains accessible in less dominant languages. This paper presents a pioneering approach to machine unlearning for multilingual language models, selectively erasing information across different languages while maintaining overall performance. Specifically, our method employs an adaptive unlearning scheme that assigns language-dependent weights to address different language performances of multilingual language models. Empirical results demonstrate the effectiveness of our framework compared to existing unlearning baselines, setting a new standard for secure and adaptable multilingual language models.