What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

TL;DR

This paper introduces a data selection framework for instruction tuning based on weighted in-context influence, improving data efficiency and effectiveness by selecting samples that reduce instruction-following difficulty.

Contribution

It proposes a novel in-context influence measure for selecting high-quality instruction data, demonstrating improved performance over baselines across models and benchmarks.

Findings

In-context influence correlates negatively with sample difficulty.

The proposed method outperforms existing baselines under limited data budgets.

Sample difficulty is not a good indicator of in-context influence.

Abstract

Instruction-tuning datasets often contain substantial redundancy and low-quality samples, necessitating effective data selection methods. We propose an instruction data selection framework based on weighted in-context influence (wICI), which measures how effectively each candidate example reduces instruction-following difficulty for semantically related peers. Through systematic experiments, we address three key questions: what constitutes effective instruction tuning data from an in-context perspective, whether sample difficulty correlates with in-context influence, and how in-context influence translates to instruction tuning effectiveness. Experiments across multiple models and benchmarks demonstrate that our method consistently outperforms existing baselines under constrained data budgets, while empirically showing that sample difficulty negatively correlates with in-context influence.