Specialist or Generalist? Instruction Tuning for Specific NLP Tasks

TL;DR

This paper explores how broad-coverage instruction tuning can enhance large language models for specific NLP tasks, especially when task data is limited, but may hinder factual tasks due to hallucinations.

Contribution

It systematically investigates the impact of generalist instruction tuning on developing specialist models across diverse tasks and provides practical guidelines for effective application.

Findings

Generalist instruction tuning improves performance on broad coverage tasks.

Limited task-specific data benefits more from generalist tuning.

Hallucinatory data can negatively impact factual knowledge tasks.

Abstract

The potential of large language models (LLMs) to simultaneously perform a wide range of natural language processing (NLP) tasks has been the subject of extensive research. Although instruction tuning has proven to be a data-efficient method for transforming LLMs into such generalist models, their performance still lags behind specialist models trained exclusively for specific tasks. In this paper, we investigate whether incorporating broad-coverage generalist instruction tuning can contribute to building a specialist model. We hypothesize that its efficacy depends on task specificity and skill requirements. Our experiments assess four target tasks with distinct coverage levels, revealing that integrating generalist instruction tuning consistently enhances model performance when the task coverage is broad. The effect is particularly pronounced when the amount of task-specific training data is limited. Further investigation into three target tasks focusing on different capabilities demonstrates that generalist instruction tuning improves understanding and reasoning abilities. However, for tasks requiring factual knowledge, generalist data containing hallucinatory information may negatively affect the model's performance. Overall, our work provides a systematic guide for developing specialist models with general instruction tuning. Our code and other related resources can be found at https://github.com/DavidFanzz/Generalist_or_Specialist.