Conflict-Resolving and Sharpness-Aware Minimization for Generalized Knowledge Editing with Multiple Updates

TL;DR

This paper introduces CoRSA, a training framework that improves knowledge editing in large language models by enhancing generalization, stability, and conflict resolution across multiple updates, outperforming existing methods.

Contribution

CoRSA is a novel, holistic training approach that simultaneously addresses generalization, stability, and conflict resolution in knowledge editing for LLMs with multiple updates.

Findings

Achieves 12.42% improvement over LoRA in fact editing benchmarks.

Reduces catastrophic forgetting by 27.82% with multiple updates.

Outperforms baselines by 5.48% Pass@5 in code domain editing.

Abstract

Large language models (LLMs) rely on internal knowledge to solve many downstream tasks, making it crucial to keep them up to date. Since full retraining is expensive, prior work has explored efficient alternatives such as model editing and parameter-efficient fine-tuning. However, these approaches often break down in practice due to poor generalization across inputs, limited stability, and knowledge conflict. To address these limitations, we propose the CoRSA (Conflict-Resolving and Sharpness-Aware Minimization) training framework, a parameter-efficient, holistic approach for knowledge editing with multiple updates. CoRSA tackles multiple challenges simultaneously: it improves generalization to different input forms and enhances stability across multiple updates by minimizing loss curvature, and resolves conflicts by maximizing the margin between new and prior knowledge. Across three widely used fact editing benchmarks, CoRSA achieves significant gains in generalization, outperforming baselines with average absolute improvements of 12.42% over LoRA and 10% over model editing methods. With multiple updates, it maintains high update efficacy while reducing catastrophic forgetting by 27.82% compared to LoRA. CoRSA also generalizes to the code domain, outperforming the strongest baseline by 5.48% Pass@5 in update efficacy.