CKnowEdit: A New Chinese Knowledge Editing Dataset for Linguistics, Facts, and Logic Error Correction in LLMs

TL;DR

This paper introduces CKnowEdit, a comprehensive Chinese knowledge editing dataset aimed at improving LLMs' ability to correct linguistic, factual, and logical errors specific to Chinese language and culture.

Contribution

We present the first Chinese knowledge editing dataset, CKnowEdit, tailored for correcting errors in LLMs across linguistic, factual, and logical domains, considering Chinese language complexities.

Findings

Current LLMs struggle with Chinese-specific linguistic and logical structures.

Evaluation of editing techniques reveals gaps in correcting Chinese knowledge errors.

The dataset enables targeted training to enhance Chinese language understanding in LLMs.

Abstract

Chinese, as a linguistic system rich in depth and complexity, is characterized by distinctive elements such as ancient poetry, proverbs, idioms, and other cultural constructs. However, current Large Language Models (LLMs) face limitations in these specialized domains, highlighting the need for the development of comprehensive datasets that can assess, continuously update, and progressively improve these culturally-grounded linguistic competencies through targeted training optimizations. To address this gap, we introduce CKnowEdit, the first-ever Chinese knowledge editing dataset designed to correct linguistic, factual, and logical errors in LLMs. We collect seven types of knowledge from a wide range of sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, taking into account the unique polyphony, antithesis, and logical structures inherent in the Chinese language. By analyzing this dataset, we highlight the challenges current LLMs face in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques reveals opportunities to advance the correction of Chinese knowledge. Code and dataset are available at https://github.com/zjunlp/EasyEdit.