Context-Robust Knowledge Editing for Language Models

TL;DR

This paper introduces CHED, a benchmark for evaluating context robustness in knowledge editing of language models, and proposes CoRE, a method that enhances editing success amidst preceding contexts while maintaining model capabilities.

Contribution

The paper develops CHED for assessing context robustness in KE and proposes CoRE to improve editing success in context-rich scenarios, addressing a key limitation of existing methods.

Findings

KE methods often fail with preceding contexts

CoRE improves editing success in context scenarios

Analysis reveals how context influences editing effectiveness

Abstract

Knowledge editing (KE) methods offer an efficient way to modify knowledge in large language models. Current KE evaluations typically assess editing success by considering only the edited knowledge without any preceding contexts. In real-world applications, however, preceding contexts often trigger the retrieval of the original knowledge and undermine the intended edit. To address this issue, we develop CHED -- a benchmark designed to evaluate the context robustness of KE methods. Evaluations on CHED show that they often fail when preceding contexts are present. To mitigate this shortcoming, we introduce CoRE, a KE method designed to strengthen context robustness by minimizing context-sensitive variance in hidden states of the model for edited knowledge. This method not only improves the editing success rate in situations where a preceding context is present but also preserves the overall capabilities of the model. We provide an in-depth analysis of the differing impacts of preceding contexts when introduced as user utterances versus assistant responses, and we dissect attention-score patterns to assess how specific tokens influence editing success.