Lifelong Knowledge Editing requires Better Regularization

TL;DR

This paper identifies causes of model degradation during sequential knowledge editing and introduces regularization techniques that enable scalable, efficient lifelong editing of large language models with reduced degradation.

Contribution

The paper formalizes locate-then-edit methods as a two-step process and introduces regularization techniques to mitigate degradation, enabling scalable and efficient lifelong knowledge editing.

Findings

Regularization reduces model degradation during editing.

Scaling to 10,000 edits with 42-61% less time.

Effective mitigation of over-optimization and norm-growth.

Abstract

Knowledge editing is a promising way to improve factuality in large language models, but recent studies have shown significant model degradation during sequential editing. In this paper, we formalize the popular locate-then-edit methods as a two-step fine-tuning process, allowing us to precisely identify the root cause of this degradation. We show that model degradation occurs due to (1) over-optimization of internal activations and (2) continuous norm-growth of edited matrices. To mitigate these issues, we introduce two regularization techniques: (1) Most-Probable Early Stopping (MPES) and (2) explicit Frobenius norm-constraint. We demonstrate that applying these simple yet effective regularization techniques at key points in the editing process can substantially mitigate model degradation. Combining these regularization methods enables scaling locate-then-edit methods to 10,000 edits while reducing editing time by 42-61%. These results show that targeted regularization is essential for lifelong knowledge editing.