An Information-Theoretic Framework for Robust Large Language Model Editing

TL;DR

This paper introduces an information-theoretic framework and a novel model editing method called IBKE, which enables precise, robust, and generalizable updates to large language models without extensive retraining.

Contribution

The paper presents a new information bottleneck-based framework and IBKE method for effective, generalizable, and minimally disruptive large language model editing.

Findings

IBKE achieves state-of-the-art accuracy in model editing tasks.

The framework improves generality and specificity of knowledge edits.

Validated across multiple LLM architectures and benchmarks.

Abstract

Large Language Models (LLMs) have become indispensable tools in science, technology, and society, enabling transformative advances across diverse fields. However, errors or outdated information within these models can undermine their accuracy and restrict their safe deployment. Developing efficient strategies for updating model knowledge without the expense and disruption of full retraining remains a critical challenge. Current model editing techniques frequently struggle to generalize corrections beyond narrow domains, leading to unintended consequences and limiting their practical impact. Here, we introduce a novel framework for editing LLMs, grounded in information bottleneck theory. This approach precisely compresses and isolates the essential information required for generalizable knowledge correction while minimizing disruption to unrelated model behaviors. Building upon this foundation, we present the Information Bottleneck Knowledge Editor (IBKE), which leverages compact latent representations to guide gradient-based updates, enabling robust and broadly applicable model editing. We validate IBKE's effectiveness across multiple LLM architectures and standard benchmark tasks, demonstrating state-of-the-art accuracy and improved generality and specificity of edits. These findings establish a theoretically principled and practical paradigm for open-domain knowledge editing, advancing the utility and trustworthiness of LLMs in real-world applications.