Benchmarking and Rethinking Knowledge Editing for Large Language Models

TL;DR

This paper conducts a comprehensive benchmarking of knowledge editing methods for large language models, revealing the limitations of parameter-based approaches and highlighting the robustness of context-based reasoning, especially in multi-edit scenarios.

Contribution

It introduces new complex datasets, evaluates multiple editing methods under realistic settings, and demonstrates the superiority of a simple context-based baseline over existing techniques.

Findings

Parameter-based methods perform poorly in realistic conditions.

The SCR baseline consistently outperforms recent methods.

Context-based reasoning shows greater robustness in multi-edit scenarios.

Abstract

Knowledge editing aims to update the embedded knowledge within Large Language Models (LLMs). However, existing approaches, whether through parameter modification or external memory integration, often suffer from inconsistent evaluation objectives and experimental setups. To address this gap, we conduct a comprehensive benchmarking study. In addition to fact-level datasets, we introduce more complex event-based datasets and general-purpose datasets drawn from other tasks. Our evaluation covers both instruction-tuned and reasoning-oriented LLMs, under a realistic autoregressive inference setting rather than teacher-forced decoding. Beyond single-edit assessments, we also evaluate multi-edit scenarios to better reflect practical demands. We employ four evaluation dimensions, including portability, and compare all recent methods against a simple and straightforward baseline named Selective Contextual Reasoning (SCR). Empirical results reveal that parameter-based editing methods perform poorly under realistic conditions. In contrast, SCR consistently outperforms them across all settings. This study offers new insights into the limitations of current knowledge editing methods and highlights the potential of context-based reasoning as a more robust alternative.