MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation

TL;DR

This paper introduces MEMLA, a novel neuron-masked low-rank adaptation method for multilingual knowledge editing, along with a comprehensive benchmark dataset, to improve update propagation across languages in large language models.

Contribution

The paper presents MEMLA, a new method for multilingual knowledge editing, and introduces MKEB, a benchmark dataset for evaluating such editing across 12 languages.

Findings

MEMLA outperforms existing baselines in multilingual knowledge editing.

The method significantly improves multi-hop reasoning after editing.

Minimal impact on downstream task performance was observed.

Abstract

Knowledge editing aims to adjust the knowledge within large language models (LLMs) to prevent their responses from becoming obsolete or inaccurate. However, existing works on knowledge editing are primarily conducted in a single language, which is inadequate for multilingual language models. In this paper, we focus on multilingual knowledge editing (MKE), which requires propagating updates across multiple languages. This necessity poses a significant challenge for the task. Furthermore, the limited availability of a comprehensive dataset for MKE exacerbates this challenge, hindering progress in this area. Hence, we introduce the Multilingual Knowledge Editing Benchmark (MKEB), a novel dataset comprising 12 languages and providing a complete evaluation framework. Additionally, we propose a method that enhances Multilingual knowledge Editing with neuron-Masked Low-Rank Adaptation (MEMLA). Specifically, we identify two categories of knowledge neurons to improve editing precision. Moreover, we perform LoRA-based editing with neuron masks to efficiently modify parameters and facilitate the propagation of updates across multiple languages. Experiments demonstrate that our method outperforms existing baselines and significantly enhances the multi-hop reasoning capability of the edited model, with minimal impact on its downstream task performance. The dataset and code will be made publicly available.