EAMET: Robust Massive Model Editing via Embedding Alignment Optimization

TL;DR

EAMET introduces a novel embedding alignment approach to improve the robustness and effectiveness of large language model editing, especially at scale and in complex scenarios.

Contribution

The paper presents EAMET, a new method that aligns embeddings to enhance large language model editing reliability and scalability.

Findings

Achieves about 90% editing efficacy on 10,000 facts.

Outperforms existing editing methods across multiple models and datasets.

Demonstrates robustness in context-rich and multi-fact editing scenarios.

Abstract

Model editing techniques are essential for efficiently updating knowledge in large language models (LLMs). However, the effectiveness of existing approaches degrades in massive editing scenarios, particularly when evaluated with practical metrics. Their robustness is also limited in context-rich settings or when editing multiple facts of the same subject simultaneously. We attribute these failures to the embedding misalignment among knowledge items, which undermines editing reliability at scale. To address this, we propose EAMET (Embedding Alignment Model Editing in Transformers), which addresses this issue by aligning the space of key and residual embeddings. Extensive experiments across six LLMs and three datasets demonstrate that EAMET consistently outperforms existing methods, achieving about 90\% editing efficacy when editing 10k facts. Codes and datasets are publicly available at https://ybdai7.github.io/eamet-page/.