Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

TL;DR

LightEdit is a scalable lifelong knowledge editing framework that selectively suppresses original knowledge in large language models, improving efficiency and stability across multiple benchmarks.

Contribution

The paper introduces LightEdit, a novel method that enhances scalability and stability in lifelong knowledge editing by selective knowledge suppression and efficient retrieval.

Findings

LightEdit outperforms existing methods on ZSRE, Counterfact, and RIPE benchmarks.

It reduces training costs, enabling cost-effective adaptation to various datasets.

The approach effectively suppresses original knowledge to facilitate accurate edits.

Abstract

Large language models (LLMs) require frequent knowledge updates to reflect changing facts and mitigate hallucinations. To meet this demand, lifelong knowledge editing has emerged as a continual approach to modify specific pieces of knowledge without retraining the entire model. Existing parameter editing methods struggle with stability during sequential edits due to catastrophic forgetting. While retrieval-based approaches are proposed to alleviate this issue, their applicability remains limited across various datasets because of high training costs. To address these limitations and enhance scalability in lifelong settings, we propose LightEdit. Our framework first selects relevant knowledge from retrieved information to modify the query effectively. It then incorporates a decoding strategy to suppress the model's original knowledge probabilities, thereby enabling efficient edits based on the selected information. Extensive experiments on ZSRE, Counterfact, and RIPE benchmarks demonstrate that LightEdit outperforms existing lifelong knowledge editing methods. Furthermore, by minimizing training costs, LightEdit achieves cost-effective scalability, enabling easy adaptation to various datasets.