COMPKE: Complex Question Answering under Knowledge Editing
Keyuan Cheng, Zijian Kan, Zhixian He, Zhuoran Zhang, Muhammad Asif Ali, Ke Xu, Lijie Hu, Di Wang

TL;DR
This paper introduces COMPKE, a new benchmark with 11,924 complex questions to evaluate knowledge editing methods in large language models, emphasizing real-life reasoning scenarios.
Contribution
The paper presents COMPKE, a benchmark specifically designed to assess knowledge editing effectiveness in complex, real-world reasoning tasks, filling a gap in existing evaluation methods.
Findings
Knowledge editing effectiveness varies significantly across models.
Current methods perform poorly on complex reasoning questions.
Evaluation reveals disparities in method performance depending on the model.
Abstract
Knowledge Editing, which efficiently modifies the knowledge in large language models, has gathered great attention. Current benchmarks primarily use multi-hop question answering to assess and analyze newly injected or updated knowledge. However, we argue that these benchmarks fail to effectively evaluate how well the updated models apply this knowledge in real-life scenarios, particularly when questions require complex reasoning, involving one-to-many relationships or multi-step logical intersections. To fill in this gap, we introduce a new benchmark, COMPKE: Complex Question Answering under Knowledge Editing, which includes 11,924 complex questions that reflect real-life situations. We conduct an extensive evaluation of four knowledge editing methods on COMPKE, revealing that their effectiveness varies notably across different models. For instance, MeLLo attains an accuracy of 39.47 on…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSemantic Web and Ontologies · Topic Modeling · Natural Language Processing Techniques
