Understanding Finetuning for Factual Knowledge Extraction

TL;DR

This paper investigates how the choice of fine-tuning data affects a model's ability to accurately recall factual knowledge, revealing that fine-tuning on lesser-known facts can harm factuality.

Contribution

It provides a theoretical explanation and empirical evidence that fine-tuning on lesser-known facts can degrade a model's factual accuracy, emphasizing the importance of data selection.

Findings

Fine-tuning on lesser-known facts reduces factual accuracy by 5-10%.

Using better-known facts for fine-tuning can match or outperform full dataset fine-tuning.

Theoretical analysis shows models may ignore subject entities when trained on lesser-known facts.

Abstract

In this work, we study the impact of QA fine-tuning data on downstream factuality. We show that fine-tuning on lesser-known facts that are poorly stored during pretraining yields significantly worse factuality than fine-tuning on well-known facts, even when all facts are seen during pretraining. We prove this phenomenon theoretically, showing that training on lesser-known facts can lead the model to ignore subject entity names and instead output a generic plausible response even when the relevant factual knowledge is encoded in the model. On three question answering benchmarks (PopQA, Entity Questions, and MMLU) and two language models (Llama-2-7B and Mistral-7B), we find that (i) finetuning on a completely factual but lesser-known subset of the data deteriorates downstream factuality (5-10%) and (ii) finetuning on a subset of better-known examples matches or outperforms finetuning on the entire dataset. Ultimately, our results shed light on the interaction between pretrained knowledge and finetuning data and demonstrate the importance of taking into account how facts are stored in the pretrained model when fine-tuning for knowledge-intensive tasks.