History Matters: Temporal Knowledge Editing in Large Language Model

TL;DR

This paper introduces Temporal Knowledge Editing (TKE), a new task for updating models with historical and current knowledge, and proposes METO, a framework that preserves historical knowledge while integrating new information.

Contribution

The paper defines TKE, creates the AToKe benchmark, and proposes METO, a novel framework that improves knowledge editing by maintaining historical facts alongside new updates.

Findings

Existing methods forget historical knowledge after editing.

METO effectively preserves historical knowledge while updating models.

AToKe benchmark reveals challenges in current model editing techniques.

Abstract

The imperative task of revising or updating the knowledge stored within large language models arises from two distinct sources: intrinsic errors inherent in the model which should be corrected and outdated knowledge due to external shifts in the real world which should be updated. Prevailing efforts in model editing conflate these two distinct categories of edits arising from distinct reasons and directly modify the original knowledge in models into new knowledge. However, we argue that preserving the model's original knowledge remains pertinent. Specifically, if a model's knowledge becomes outdated due to evolving worldly dynamics, it should retain recollection of the historical knowledge while integrating the newfound knowledge. In this work, we introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe (Assessment of TempOral Knowledge Editing) to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models, which edits both historical and new knowledge concurrently and optimizes the model's prediction for the time of each fact. Our assessments demonstrate that while AToKe is still difficult, METO maintains the effectiveness of learning new knowledge and meanwhile substantially improves the performance of edited models on utilizing historical knowledge.