Efficient Knowledge Injection in LLMs via Self-Distillation

TL;DR

This paper introduces a prompt distillation method for efficiently injecting new factual knowledge into large language models, outperforming traditional fine-tuning and matching retrieval-augmented generation without needing larger models or structured data.

Contribution

It presents a novel self-distillation approach for knowledge injection that is simpler and more effective than existing fine-tuning and RAG methods.

Findings

Prompt distillation outperforms supervised fine-tuning.

It can surpass retrieval-augmented generation in knowledge injection.

The method scales effectively across multiple LLM sizes and families.

Abstract

In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.