Are We Evaluating the Edit Locality of LLM Model Editing Properly?

TL;DR

This paper critically examines current evaluation methods for assessing the specificity of model edits in large language models, identifies their shortcomings, and proposes a new, more sensitive evaluation protocol that better correlates with regularizer strength.

Contribution

It introduces a novel evaluation protocol for model editing specificity that addresses conceptual and empirical limitations of existing metrics.

Findings

Existing metrics are weakly correlated with regularizer strength.

Current metrics lack sensitivity to distinguish different methods.

The proposed protocol improves sensitivity and correlation with regularizer strength.

Abstract

Model editing has recently emerged as a popular paradigm for efficiently updating knowledge in LLMs. A central desideratum of updating knowledge is to balance editing efficacy, i.e., the successful injection of target knowledge, and specificity (also known as edit locality), i.e., the preservation of existing non-target knowledge. However, we find that existing specificity evaluation protocols are inadequate for this purpose. We systematically elaborated on the three fundamental issues it faces. Beyond the conceptual issues, we further empirically demonstrate that existing specificity metrics are weakly correlated with the strength of specificity regularizers. We also find that current metrics lack sufficient sensitivity, rendering them ineffective at distinguishing the specificity performance of different methods. Finally, we propose a constructive evaluation protocol. Under this protocol, the conflict between open-ended LLMs and the assumption of determined answers is eliminated, query-independent fluency biases are avoided, and the evaluation strictness can be smoothly adjusted within a near-continuous space. Experiments across various LLMs, datasets, and editing methods show that metrics derived from the proposed protocol are more sensitive to changes in the strength of specificity regularizers and exhibit strong correlation with them, enabling more fine-grained discrimination of different methods' knowledge preservation capabilities.