Beyond Perplexity: A Lightweight Benchmark for Knowledge Retention in Supervised Fine-Tuning

TL;DR

This paper introduces KR-Test, a lightweight evaluation method that effectively measures factual knowledge retention in LLMs during supervised fine-tuning, surpassing traditional perplexity metrics.

Contribution

It proposes a novel, corpus-grounded benchmark for assessing knowledge retention, enabling better understanding of fine-tuning effects on factual learning.

Findings

KR-Test accurately distinguishes factual learning from stylistic mimicry.

The framework reveals the dynamics of knowledge retention during training.

Analysis of LoRA shows dissociation between linguistic convergence and factual retention.

Abstract

Supervised Fine-Tuning (SFT) is a standard approach for injecting domain knowledge into Large Language Models (LLMs). However, relying on validation perplexity to monitor training is often insufficient, as it confounds stylistic mimicry with genuine factual internalization. To address this, we introduce the Knowledge Retention (KR) Test , a lightweight, corpus-grounded evaluation framework designed to distinguish factual learning from linguistics. KR-Test utilizes automatically generated contrastive examples to measure likelihood preferences for correct versus incorrect continuations, requiring no instruction tuning or generative decoding. We validate the framework's integrity through a "blind vs. oracle" baseline analysis. Furthermore, we demonstrate the diagnostic capabilities of KR-Test by analyzing the training dynamics of Low-Rank Adaptation (LoRA). By exposing the fine-grained dissociation between linguistic convergence and knowledge retention, KR-Test enhances the interpretability of fine-tuning dynamics.