Knowledge Editing in Language Models via Adapted Direct Preference Optimization

TL;DR

This paper introduces Knowledge Direct Preference Optimization (KDPO), a novel method for efficiently editing knowledge in large language models by continually updating their stored information without retraining.

Contribution

The paper presents KDPO, a new online approach for knowledge editing in LLMs that improves localization and effectiveness of knowledge updates compared to existing methods.

Findings

KDPO achieves comparable or better performance than previous methods.

It effectively handles multiple sequential knowledge edits.

Ablation studies confirm the advantages of KDPO over standard DPO.

Abstract

Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods.