SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe

TL;DR

SFTMix introduces a Mixup-based regularization method for instruction tuning of large language models, improving performance without requiring high-quality datasets by interpolating examples based on confidence levels.

Contribution

The paper proposes SFTMix, a novel Mixup recipe that leverages training dynamics to interpolate examples with different confidence levels, enhancing instruction tuning without curated datasets.

Findings

SFTMix improves instruction-following performance across multiple LLMs.

SFTMix enhances healthcare-specific SFT tasks.

The method is compatible with data selection and scalable to various applications.

Abstract

To acquire instruction-following capabilities, large language models (LLMs) undergo instruction tuning, where they are trained on instruction-response pairs using next-token prediction (NTP). Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning (SFT) datasets, typically requiring data filtering with proprietary LLMs or human annotation. In this paper, we take a different approach by proposing SFTMix, a novel Mixup-based recipe that elevates LLM instruction tuning without relying on well-curated datasets. We observe that LLMs exhibit uneven confidence across the semantic representation space. We argue that examples with different confidence levels should play distinct roles in instruction tuning: Confident data is prone to overfitting, while unconfident data is harder to generalize. Based on this insight, SFTMix leverages training dynamics to identify examples with varying confidence levels. We then interpolate them to bridge the confidence gap and apply a Mixup-based regularization to support learning on these additional, interpolated examples. We demonstrate the effectiveness of SFTMix in both instruction-following and healthcare-specific SFT tasks, with consistent improvements across LLM families and SFT datasets of varying sizes and qualities. Extensive analyses across six directions highlight SFTMix's compatibility with data selection, adaptability to compute-constrained scenarios, and scalability to broader applications.