Aligning Language Models with Real-time Knowledge Editing

TL;DR

This paper introduces CRAFT, a dynamic dataset for real-time knowledge editing in language models, and proposes KEDAS, a new method that improves editing performance and adaptability.

Contribution

The work presents a novel dataset and a new paradigm for knowledge editing, enabling models to adapt to evolving real-world information more effectively.

Findings

CRAFT dataset enables comprehensive evaluation of knowledge editing methods.

KEDAS significantly outperforms previous methods on CRAFT and traditional datasets.

Proposes a shift from static to dynamic knowledge editing in language models.

Abstract

Knowledge editing aims to modify outdated knowledge in language models efficiently while retaining their original capabilities. Mainstream datasets for knowledge editing are predominantly static and fail to keep in pace with the evolving real-world knowledge. In this work, we introduce CRAFT, an ever-evolving real-world dataset for knowledge editing. It evaluates models on temporal locality, common-sense locality, composite portability and alias portability, providing a comprehensive and challenging evaluation for knowledge editing, on which previous methods hardly achieve balanced performance. Towards flexible real-time knowledge editing, we propose KEDAS, a novel paradigm of knowledge editing alignment featuring diverse edit augmentation and self-adaptive post-alignment inference, exhibiting significant performance gain on both CRAFT and traditional datasets compared to previous methods. We hope this work may serve as a catalyst for shifting the focus of knowledge editing from static update to dynamic evolution.