MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs

TL;DR

MEMIT-Merge improves large language model knowledge editing by resolving key-value conflicts in batch edits sharing the same subject, significantly enhancing success rates in such scenarios.

Contribution

It introduces MEMIT-Merge, a novel method that merges value computations to address conflicts in same-subject batch editing, outperforming MEMIT.

Findings

MEMIT-Merge maintains over 90% success rate in large batch edits.

Standard MEMIT's success rate drops to around 50% with larger batches.

MEMIT-Merge effectively resolves key-value conflicts in same-subject batch editing.

Abstract

As large language models continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncover that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals this stems from MEMIT's key value modeling framework: identical keys (derived from the shared subject) are forced to represent different values (corresponding to different knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in samesubject batch editing scenarios. Experimental results demonstrate that when MEMIT's edit success rate drops to around 50% at larger batch sizes, MEMIT-Merge maintains a success rate exceeding 90%, showcasing remarkable robustness to subject entity collisions. The code is available at https://github.com/NUSTM/ MEMIT-Merge.