Finetuned Language Models Are Zero-Shot Learners

TL;DR

This paper demonstrates that instruction tuning significantly enhances the zero-shot learning capabilities of large language models, outperforming GPT-3 on various NLP tasks by finetuning on diverse instruction-based datasets.

Contribution

The authors introduce FLAN, a method of instruction tuning large language models on multiple tasks, leading to substantial improvements in zero-shot and few-shot performance.

Findings

FLAN surpasses zero-shot GPT-3 on 20 of 25 tasks.

Instruction tuning improves performance more than increasing model size alone.

Number of datasets and natural language instructions are crucial for success.

Abstract

This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection of tasks described via instructions -- substantially improves zero-shot performance on unseen tasks. We take a 137B parameter pretrained language model and instruction-tune it on over 60 NLP tasks verbalized via natural language instruction templates. We evaluate this instruction-tuned model, which we call FLAN, on unseen task types. FLAN substantially improves the performance of its unmodified counterpart and surpasses zero-shot 175B GPT-3 on 20 of 25 tasks that we evaluate. FLAN even outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze. Ablation studies reveal that number of finetuning datasets, model scale, and natural language instructions are key to the success of instruction tuning.