UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models

TL;DR

UniEdit is a comprehensive benchmark for evaluating large language model editing across diverse open-domain knowledge, using a novel sampling algorithm and natural language conversion to assess ripple effects and editing performance.

Contribution

The paper introduces UniEdit, a unified, open-domain knowledge editing benchmark with a new sampling algorithm and natural language conversion, addressing limitations of existing datasets.

Findings

Extensive coverage across 25 domains and 5 categories.

Effective evaluation of ripple effects through NMCS sampling.

Insights into LLM editing strengths and weaknesses.

Abstract

Model editing aims to enhance the accuracy and reliability of large language models (LLMs) by efficiently adjusting their internal parameters. Currently, most LLM editing datasets are confined to narrow knowledge domains and cover a limited range of editing evaluation. They often overlook the broad scope of editing demands and the diversity of ripple effects resulting from edits. In this context, we introduce UniEdit, a unified benchmark for LLM editing grounded in open-domain knowledge. First, we construct editing samples by selecting entities from 25 common domains across five major categories, utilizing the extensive triple knowledge available in open-domain knowledge graphs to ensure comprehensive coverage of the knowledge domains. To address the issues of generality and locality in editing, we design an Neighborhood Multi-hop Chain Sampling (NMCS) algorithm to sample subgraphs based on a given knowledge piece to entail comprehensive ripple effects to evaluate. Finally, we employ proprietary LLMs to convert the sampled knowledge subgraphs into natural language text, guaranteeing grammatical accuracy and syntactical diversity. Extensive statistical analysis confirms the scale, comprehensiveness, and diversity of our UniEdit benchmark. We conduct comprehensive experiments across multiple LLMs and editors, analyzing their performance to highlight strengths and weaknesses in editing across open knowledge domains and various evaluation criteria, thereby offering valuable insights for future research endeavors.