Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality

TL;DR

This study systematically investigates how data choices, layer modifications, and training strategies influence the alignment quality of large language models through extensive supervised fine-tuning experiments.

Contribution

It provides a comprehensive analysis of factors affecting LLM fine-tuning, revealing key dataset properties, layer-wise changes, and the predictive power of perplexity for SFT success.

Findings

Perplexity reliably predicts SFT effectiveness.

Layer-wise weight changes correlate with performance improvements.

Certain training-task synergies are consistent across models.

Abstract

Supervised fine-tuning (SFT) is a critical step in aligning large language models (LLMs) with human instructions and values, yet many aspects of SFT remain poorly understood. We trained a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks, resulting in 1,000+ SFT models under controlled conditions. We then identified the dataset properties that matter most and examined the layer-wise modifications introduced by SFT. Our findings reveal that some training-task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies. Moreover, we demonstrate that perplexity consistently predicts SFT effectiveness, often surpassing superficial similarity between the training data and the benchmark, and that mid-layer weight changes correlate most strongly with performance gains. We release these 1,000+ SFT models and benchmark results to accelerate further research. All resources are available at https://github.com/llm-jp/massive-sft.