ScEdit: Script-based Assessment of Knowledge Editing

TL;DR

This paper introduces ScEdit, a comprehensive script-based benchmark for evaluating knowledge editing methods in language models, highlighting their challenges in real-world scenarios and across different evaluation metrics.

Contribution

The paper presents a novel benchmark, ScEdit, that extends traditional fact-based evaluation to action-based tasks and integrates multiple evaluation methods for comprehensive analysis.

Findings

All KE methods show performance drops on established metrics.

KE methods face challenges on text-level evaluation metrics.

The benchmark reveals the difficulty of real-world knowledge editing tasks.

Abstract

Knowledge Editing (KE) has gained increasing attention, yet current KE tasks remain relatively simple. Under current evaluation frameworks, many editing methods achieve exceptionally high scores, sometimes nearing perfection. However, few studies integrate KE into real-world application scenarios (e.g., recent interest in LLM-as-agent). To support our analysis, we introduce a novel script-based benchmark -- ScEdit (Script-based Knowledge Editing Benchmark) -- which encompasses both counterfactual and temporal edits. We integrate token-level and text-level evaluation methods, comprehensively analyzing existing KE techniques. The benchmark extends traditional fact-based ("What"-type question) evaluation to action-based ("How"-type question) evaluation. We observe that all KE methods exhibit a drop in performance on established metrics and face challenges on text-level metrics, indicating a challenging task. Our benchmark is available at https://github.com/asdfo123/ScEdit.