AnyEdit: Edit Any Knowledge Encoded in Language Models

TL;DR

AnyEdit introduces a novel autoregressive method for updating long-form and diverse knowledge in language models, overcoming previous limitations by sequentially editing chunks and achieving significant performance improvements.

Contribution

It presents a new editing paradigm that decomposes knowledge into chunks and iteratively updates them, enabling effective editing of long and diverse knowledge in LLMs.

Findings

Outperforms baselines by 21.5% on key benchmarks.

Effectively updates knowledge across various formats like poetry, code, and math.

Serves as a plug-and-play framework for existing editing methods.

Abstract

Large language models (LLMs) often produce incorrect or outdated information, necessitating efficient and precise knowledge updates. Current model editing methods, however, struggle with long-form knowledge in diverse formats, such as poetry, code snippets, and mathematical derivations. These limitations arise from their reliance on editing a single token's hidden state, a limitation we term "efficacy barrier". To solve this, we propose AnyEdit, a new autoregressive editing paradigm. It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. Theoretically, we ground AnyEdit in the Chain Rule of Mutual Information, showing its ability to update any knowledge within LLMs. Empirically, it outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge. Additionally, AnyEdit serves as a plug-and-play framework, enabling current editing methods to update knowledge with arbitrary length and format, significantly advancing the scope and practicality of LLM knowledge editing.