Commonsense Knowledge Editing Based on Free-Text in LLMs

TL;DR

This paper introduces a novel approach for editing commonsense knowledge in large language models using free-text, addressing limitations of previous methods that focused on single tokens or entities.

Contribution

It proposes the Knowledge Localization for Free-Text (KLFT) method and the Dynamics-aware Editing Method (DEM), enabling effective editing of broad, long, and non-instantiated commonsense knowledge.

Findings

DEM achieves excellent editing performance.

KLFT reveals distribution challenges of knowledge in model layers.

The approach effectively updates commonsense knowledge in LLMs.

Abstract

Knowledge editing technology is crucial for maintaining the accuracy and timeliness of large language models (LLMs) . However, the setting of this task overlooks a significant portion of commonsense knowledge based on free-text in the real world, characterized by broad knowledge scope, long content and non instantiation. The editing objects of previous methods (e.g., MEMIT) were single token or entity, which were not suitable for commonsense knowledge in free-text form. To address the aforementioned challenges, we conducted experiments from two perspectives: knowledge localization and knowledge editing. Firstly, we introduced Knowledge Localization for Free-Text(KLFT) method, revealing the challenges associated with the distribution of commonsense knowledge in MLP and Attention layers, as well as in decentralized distribution. Next, we propose a Dynamics-aware Editing Method(DEM), which utilizes a Dynamics-aware Module to locate the parameter positions corresponding to commonsense knowledge, and uses Knowledge Editing Module to update knowledge. The DEM method fully explores the potential of the MLP and Attention layers, and successfully edits commonsense knowledge based on free-text. The experimental results indicate that the DEM can achieve excellent editing performance.