Learning and Unlearning of Fabricated Knowledge in Language Models

TL;DR

This paper investigates how injected facts into language models are retained or forgotten over training, revealing a sweet spot of fact novelty that affects memory longevity and model hallucinations, with implications for data poisoning mitigation.

Contribution

It introduces a new probing dataset 'Outlandish' and demonstrates how different types of injected facts influence memory retention and hallucination, proposing a simple method to erase conflicting knowledge.

Findings

Conflicting facts are retained for tens of thousands of training steps.

Mundane and scrambled prompts are forgotten more rapidly.

Multi-step sparse updates can largely erase conflicting knowledge.

Abstract

What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this question by injecting facts into LMs from a new probing dataset, "Outlandish", which is designed to permit the testing of a spectrum of different fact types. When studying how robust these memories are, there appears to be a sweet spot in the spectrum of fact novelty between consistency with world knowledge and total randomness, where the injected memory is the most enduring. Specifically we show that facts that conflict with common knowledge are remembered for tens of thousands of training steps, while prompts not conflicting with common knowledge (mundane), as well as scrambled prompts (randomly jumbled) are both forgotten much more rapidly. Further, knowledge-conflicting facts can "prime'' how the language model hallucinates on logically unrelated prompts, showing their propensity for non-target generalization, while both mundane and randomly jumbled facts prime significantly less. Finally, we show that impacts of knowledge-conflicting facts in LMs, though they can be long lasting, can be largely erased by novel application of multi-step sparse updates, even while the training ability of the model is preserved. As such, this very simple procedure has direct implications for mitigating the effects of data poisoning in training.