BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

TL;DR

This paper presents BMIKE-53, a new benchmark for evaluating cross-lingual knowledge editing across 53 languages, highlighting the importance of model size, demonstrations, and linguistic properties in performance.

Contribution

It introduces BMIKE-53, a comprehensive benchmark for cross-lingual in-context knowledge editing, and systematically evaluates factors affecting performance across diverse languages.

Findings

Larger models and tailored demonstrations improve cross-lingual KE.

Script type significantly impacts performance, with non-Latin scripts underperforming.

Model scale and demonstration alignment are critical for effective cross-lingual knowledge editing.

Abstract

This paper introduces BMIKE-53, a comprehensive benchmark for cross-lingual in-context knowledge editing (IKE) across 53 languages, unifying three knowledge editing (KE) datasets: zsRE, CounterFact, and WikiFactDiff. Cross-lingual KE, which requires knowledge edited in one language to generalize across others while preserving unrelated knowledge, remains underexplored. To address this gap, we systematically evaluate IKE under zero-shot, one-shot, and few-shot setups, incorporating tailored metric-specific demonstrations. Our findings reveal that model scale and demonstration alignment critically govern cross-lingual IKE efficacy, with larger models and tailored demonstrations significantly improving performance. Linguistic properties, particularly script type, strongly influence performance variation across languages, with non-Latin languages underperforming due to issues like language confusion. Code and data are publicly available at: https://github.com/ercong21/MultiKnow/.