Diversity in Large Language Models under Supervised Fine-Tuning

TL;DR

This paper investigates how supervised fine-tuning reduces diversity in large language models and introduces TOFU loss to mitigate this effect, improving output variety without sacrificing quality.

Contribution

It provides a theoretical analysis of diversity loss in SFT and proposes a novel TOFU loss to enhance diversity while maintaining response quality.

Findings

Generation breadth narrows after SFT.

TOFU improves diversity across models and benchmarks.

TOFU preserves high response quality.

Abstract

Supervised Fine-Tuning (SFT) is essential for aligning Large Language Models (LLMs) with user intent, yet it is believed to suppress generative diversity. Although this reduction is frequently referenced, formal empirical testing of the phenomenon remains limited. The expressiveness of LLMs by itself was addressed by multiple prior methods. Their varying perspectives suggest that deeper investigation could yield further improvements. In this study, we attribute the decline to two primary drivers: the neglect of low-frequency patterns within fine-tuning datasets and the forgetting of preexisting knowledge. Motivated by our theoretical analysis, we develop Tempered Focal (TOFU) loss, a novel objective that addresses both stated challenges simultaneously. Our extensive evaluation confirms at scale that generation breadth narrows after SFT and strengthens the hypothesis explaining this effect. Across multiple models and benchmarks, we demonstrate that TOFU enhances output diversity while preserving high response quality, offering a principled approach to SFT.