Rethinking Residual Distribution in Locate-then-Edit Model Editing

TL;DR

This paper analyzes residual distribution in locate-then-edit model editing methods, identifies a key failure mode, and proposes the BLUE strategy to improve editing accuracy and preserve model capabilities, achieving significant performance gains.

Contribution

It reveals the failure mode of residual distribution in model editing and introduces the BLUE strategy to address it, improving editing precision and model capability preservation.

Findings

BLUE improves editing performance by 35.59% on average.

Residual distribution errors increase with distribution distance, batch size, and edit sequence length.

BLUE enhances the state-of-the-art in model editing across multiple LLMs and datasets.

Abstract

Model editing enables targeted updates to the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute a prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-based multi-layer updates via $\textbf{residual distribution}$. While empirically effective, we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, we show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate or suboptimal edits. To address this, we propose the $\textbf{B}$oundary $\textbf{L}$ayer $\textbf{U}$pdat$\textbf{E (BLUE)}$ strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.