Neuron-Level Sequential Editing for Large Language Models

TL;DR

This paper introduces NSE, a neuron-level sequential editing method for large language models that effectively updates internal knowledge through multiple rounds while minimizing forgetting and failure.

Contribution

The paper presents NSE, a novel neuron-level sequential editing approach that supports multi-round updates in LLMs, addressing limitations of existing single-round editing methods.

Findings

NSE outperforms existing model editing methods in experiments.

NSE effectively mitigates model forgetting during sequential editing.

NSE supports multiple rounds of knowledge updates without significant failure.

Abstract

This work explores sequential model editing in large language models (LLMs), a critical task that involves modifying internal knowledge within LLMs continuously through multi-round editing, each incorporating updates or corrections to adjust the model outputs without the need for costly retraining. Existing model editing methods, especially those that alter model parameters, typically focus on single-round editing and often face significant challenges in sequential model editing-most notably issues of model forgetting and failure. To address these challenges, we introduce a new model editing method, namely \textbf{N}euron-level \textbf{S}equential \textbf{E}diting (NSE), tailored for supporting sequential model editing. Specifically, we optimize the target layer's hidden states using the model's original weights to prevent model failure. Furthermore, we iteratively select neurons in multiple layers for editing based on their activation values to mitigate model forgetting. Our empirical experiments demonstrate that NSE significantly outperforms current modifying parameters model editing methods, marking a substantial advancement in the field of sequential model editing. Our code is released on \url{https://github.com/jianghoucheng/NSE}.