HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

TL;DR

HoReN introduces a novel codebook-based, parameter-preserving method for large-scale model editing that maintains high performance over thousands of sequential edits using Hopfield dynamics.

Contribution

The paper presents HoReN, a new memory-augmented editor that improves scalability and stability of model editing across many sequential updates.

Findings

HoReN outperforms existing editors on multiple benchmarks.

HoReN maintains over 0.93 accuracy after 50K edits.

HoReN scales effectively without degradation over extensive editing sequences.

Abstract

Large language models encode vast factual knowledge that can become outdated or incorrect after deployment, yet retraining is prohibitively costly. This motivates lifelong model editing, which updates targeted behavior while preserving the rest of the model. Existing editors, both parameter-modifying and parameter-preserving, degrade severely as edits accumulate and struggle to generalize across paraphrases. We propose HoReN, a codebook-based parameter-preserving editor that wraps a single MLP layer with a discrete key-value memory. HoReN treats each codebook entry as both a knowledge key and a Hopfield stored pattern, retrieves edits by angular similarity on the unit hypersphere, and refines queries through damped Hopfield dynamics so paraphrases converge to the correct memory basin while unrelated inputs remain stable. HoReN achieves strong editing performance with consistent gains across diverse benchmarks spanning standard ZsRE, structured WikiBigEdit, and unstructured UnKE evaluations. Moreover, HoReN scales to 50K sequential edits on ZsRE with stable overall performance above 0.93, while prior editors collapse or degrade severely before reaching 10K. Our code is available at https://github.com/ha11ucin8/HoReN.