Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks

TL;DR

This paper presents a simple, instruction-based task selection method that improves instruction tuning efficiency and effectiveness for specific tasks, outperforming traditional transferability-based approaches.

Contribution

The authors introduce an instruction-only task selection technique that enhances instruction tuning by aligning with instructional styles, leading to better task relevance and performance.

Findings

Significant performance improvements on benchmarks like P3, Big-Bench, NIV2, and Big-Bench Hard.

Outperforms prior task selection methods in efficiency and effectiveness.

Requires only instruction information, avoiding complex transferability measurements.

Abstract

Instruction tuning has been proven effective in enhancing zero-shot generalization across various tasks and in improving the performance of specific tasks. For task-specific improvements, strategically selecting and training on related tasks that provide meaningful supervision is crucial, as this approach enhances efficiency and prevents performance degradation from learning irrelevant tasks. In this light, we introduce a simple yet effective task selection method that leverages instruction information alone to identify relevant tasks, optimizing instruction tuning for specific tasks. Our method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Additionally, by aligning the model with the unique instructional template style of the meta-dataset, we enhance its ability to granularly discern relevant tasks, leading to improved overall performance. Experimental results demonstrate that training on a small set of tasks, chosen solely based on the instructions, results in substantial improvements in performance on benchmarks such as P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these improvements surpass those achieved by prior task selection methods, highlighting the superiority of our approach.