Rethinking Data Selection for Supervised Fine-Tuning

TL;DR

This paper challenges traditional data selection methods for supervised fine-tuning of language models, proposing that focusing on responses with detailed, human-like interactions improves performance more than data quality or diversity.

Contribution

It introduces a novel heuristic of selecting instances with long responses, which better captures human-like interactions for supervised fine-tuning.

Findings

Selecting long-response instances outperforms full datasets.

Long responses better reflect human-like conversation styles.

Simple heuristics can effectively improve fine-tuning outcomes.

Abstract

Although supervised finetuning (SFT) has emerged as an essential technique to align large language models with humans, it is considered superficial, with style learning being its nature. At the same time, recent works indicate the importance of data selection for SFT, showing that finetuning with high-quality and diverse subsets of the original dataset leads to superior downstream performance. In this work, we rethink the intuition behind data selection for SFT. Considering SFT is superficial, we propose that essential demonstrations for SFT should focus on reflecting human-like interactions instead of data quality or diversity. However, it is not straightforward to directly assess to what extent a demonstration reflects human styles. Towards an initial attempt in this direction, we find selecting instances with long responses is surprisingly more effective for SFT than utilizing full datasets or instances selected based on quality and diversity. We hypothesize that such a simple heuristic implicitly mimics a crucial aspect of human-style conversation: detailed responses are usually more helpful.