Can We Edit Factual Knowledge by In-Context Learning?

TL;DR

This paper investigates whether in-context learning can be used to edit factual knowledge in large language models without fine-tuning, demonstrating competitive success rates with fewer side effects across various model scales.

Contribution

The study provides a comprehensive empirical analysis of in-context knowledge editing, showing its effectiveness and scalability compared to traditional gradient-based methods.

Findings

In-context knowledge editing achieves competitive success rates on GPT-J.

IKE results in fewer over-editing and knowledge forgetting.

The method scales effectively to larger models like OPT-175B.

Abstract

Previous studies have shown that large language models (LLMs) like GPTs store massive factual knowledge in their parameters. However, the stored knowledge could be false or out-dated. Traditional knowledge editing methods refine LLMs via fine-tuning on texts containing specific knowledge. However, with the increasing scales of LLMs, these gradient-based approaches bring large computation costs. The trend of model-as-a-service also makes it impossible to modify knowledge in black-box LMs. Inspired by in-context learning (ICL), a new paradigm based on demonstration contexts without parameter updating, we explore whether ICL can edit factual knowledge. To answer this question, we give a comprehensive empirical study of ICL strategies. Experiments show that in-context knowledge editing (IKE), without any gradient and parameter updating, achieves a competitive success rate compared to gradient-based methods on GPT-J (6B) but with much fewer side effects, including less over-editing on similar but unrelated facts and less knowledge forgetting on previously stored knowledge. We also apply the method to larger LMs with tens or hundreds of parameters like OPT-175B, which shows the scalability of our method. The code is available at https://github.com/Zce1112zslx/IKE.