Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning

TL;DR

This paper investigates how to best allocate limited annotation budgets between supervised and preference-based finetuning for large language models, revealing that initial supervised training is crucial in low-data regimes and a hybrid approach benefits larger budgets.

Contribution

It provides a comprehensive analysis of budget allocation strategies between supervised and preference finetuning, offering practical guidelines for optimizing model performance under data constraints.

Findings

Supervised finetuning on base models is best in low-data regimes (<1,000 examples).

Combining SFT and PFT with increasing preference data improves performance at larger budgets.

Skipping SFT and directly applying PFT causes a cold start problem due to distribution shift.

Abstract

Post-training of Large Language Models often involves a pipeline of Supervised Finetuning (SFT) followed by Preference Finetuning (PFT) using methods like Direct Preference Optimization. Both stages require annotated data that are very different in structure and costs. We study how to optimally allocate a fixed training data budget between the two stages, through extensive experiments spanning four diverse tasks, multiple model sizes and various data annotation costs. Our findings reveal that just SFT on the base model dominates performance in low-data regimes ($<1,000$ annotated examples). With larger data-budgets, we observe that a combination of SFT and PFT, often with increasing portions allocated towards preference data yields optimal performance. However, completely eliminating SFT and running PFT directly on the base model yields suboptimal performance, described as the cold start problem on tasks like mathematics. We observe that this is due to the distribution shift arising from using DPO directly on the base model to elicit step-by-step reasoning. This limitation can be effectively addressed by allocating even a small portion ($<10$%) of the budget to SFT first, resulting in performance improvements of $15-20$% on analytical benchmarks like GSM8k. These results provide actionable insights for researchers and practitioners optimizing model development under budget constraints, where high-quality data curation often represents a significant portion of the total costs of model development.