Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision

TL;DR

This paper introduces the Task--Specificity Score (TSS) to measure how much instructions influence the output in instruction tuning, revealing that task-specific examples enhance model performance especially under limited token budgets.

Contribution

The paper proposes TSS and TSS++, novel metrics for quantifying instruction importance, and demonstrates their effectiveness across multiple datasets and language models.

Findings

TSS correlates with downstream performance improvements.

Selecting task-specific examples boosts model accuracy under token constraints.

TSS++ mitigates easy-negative effects in instruction evaluation.

Abstract

Instruction tuning is now the default way to train and adapt large language models, but many instruction--input--output pairs are only weakly specified: for a given input, the same output can remain plausible under several alternative instructions. This raises a simple question: \emph{does the instruction uniquely determine the target output?} We propose the \textbf{Task--Specificity Score (TSS)} to quantify how much an instruction matters for predicting its output, by contrasting the true instruction against plausible alternatives for the same input. We further introduce \textbf{TSS++}, which uses hard alternatives and a small quality term to mitigate easy-negative effects. Across three instruction datasets (\textsc{Alpaca}, \textsc{Dolly-15k}, \textsc{NI-20}) and three open LLMs (Gemma, Llama, Qwen), we show that selecting task-specific examples improves downstream performance under tight token budgets and complements quality-based filters such as perplexity and IFD.