NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV Database

TL;DR

NeuralDB introduces a neural key-value database framework for large-scale knowledge editing in LLMs, enabling efficient updates of up to 100,000 facts while preserving model performance.

Contribution

The paper proposes NeuralDB, a novel neural KV database with a gated retrieval module, significantly scaling knowledge editing capabilities in LLMs.

Findings

Outperforms existing methods in editing efficacy and generalization.

Maintains overall task performance after large-scale edits.

Effective even when scaling to 100,000 facts.

Abstract

Efficiently editing knowledge stored in large language models (LLMs) enables model updates without large-scale training. One possible solution is Locate-and-Edit (L\&E), allowing simultaneous modifications of a massive number of facts. However, such editing may compromise the general abilities of LLMs and even result in forgetting edited facts when scaling up to thousands of edits. In this paper, we model existing linear L\&E methods as querying a Key-Value (KV) database. From this perspective, we then propose NeuralDB, an editing framework that explicitly represents the edited facts as a neural KV database equipped with a non-linear gated retrieval module, % In particular, our gated module only operates when inference involves the edited facts, effectively preserving the general abilities of LLMs. Comprehensive experiments involving the editing of 10,000 facts were conducted on the ZsRE and CounterFacts datasets, using GPT2-XL, GPT-J (6B) and Llama-3 (8B). The results demonstrate that NeuralDB not only excels in editing efficacy, generalization, specificity, fluency, and consistency, but also preserves overall performance across six representative text understanding and generation tasks. Further experiments indicate that NeuralDB maintains its effectiveness even when scaled to 100,000 facts (\textbf{50x} more than in prior work).