How Many Data Samples is an Additional Instruction Worth?

TL;DR

This paper investigates whether adding alternate instructions can improve instruction-tuned NLP models, finding that such augmentation can be as effective as adding around 200 data samples, especially in low-data scenarios.

Contribution

The study demonstrates that instruction augmentation with alternate instructions significantly enhances model performance, offering a practical way for non-experts to improve NLP tasks without extensive data collection.

Findings

Instruction augmentation improves performance by up to 35%.

Additional instructions can be equivalent to approximately 200 data samples.

Effectiveness is especially pronounced in low-data regimes.

Abstract

Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state-of-the-art task-specific models. Conventional approaches to improve model performance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augmentation helpful? We augment a subset of tasks in the expanded version of NATURAL INSTRUCTIONS with additional instructions and find that it significantly improves model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to ~200 data samples on average across tasks.