EMSEdit: Efficient Multi-Step Meta-Learning-based Model Editing

TL;DR

EMSEdit introduces an efficient multi-step meta-learning approach for model editing in large language models, significantly improving performance and efficiency, especially in low-data scenarios, by leveraging multi-step backpropagation and regularization techniques.

Contribution

The paper proposes EMSEdit, a novel multi-step meta-learning method that enhances model editing efficiency and effectiveness, particularly under limited data, by integrating multi-step backpropagation and norm-based regularization.

Findings

Outperforms state-of-the-art methods in sequential and batch editing tasks.

Effectively handles complex multi-hop reasoning edits.

Seamlessly integrates with existing approaches for additional gains.

Abstract

Large Language Models (LLMs) power numerous AI applications, yet updating their knowledge remains costly. Model editing provides a lightweight alternative through targeted parameter modifications, with meta-learning-based model editing (MLME) demonstrating strong effectiveness and efficiency. However, we find that MLME struggles in low-data regimes and incurs high training costs due to the use of KL divergence. To address these issues, we propose $\textbf{E}$fficient $\textbf{M}$ulti-$\textbf{S}$tep $\textbf{Edit (EMSEdit)}$, which leverages multi-step backpropagation (MSBP) to effectively capture gradient-activation mapping patterns within editing samples, performs multi-step edits per sample to enhance editing performance under limited data, and introduces norm-based regularization to preserve unedited knowledge while improving training efficiency. Experiments on two datasets and three LLMs show that EMSEdit consistently outperforms state-of-the-art methods in both sequential and batch editing. Moreover, MSBP can be seamlessly integrated into existing approaches to yield additional performance gains. Further experiments on a multi-hop reasoning editing task demonstrate EMSEdit's robustness in handling complex edits, while ablation studies validate the contribution of each design component. Our code is available at https://github.com/xpq-tech/emsedit.