What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

TL;DR

This study systematically compares different Transformer architectures and pretraining objectives to determine their effectiveness for zero-shot generalization, revealing that causal models excel in pure pretraining while masked models with finetuning perform best overall.

Contribution

It provides a large-scale, systematic evaluation of architecture and pretraining objectives for zero-shot generalization in text-to-text models, including adaptation strategies between architectures.

Findings

Causal decoder-only models trained with autoregressive objectives show strong zero-shot performance.

Masked language modeling with multitask finetuning yields the best overall results.

Pretrained non-causal models can be adapted into causal models and vice versa for improved performance.

Abstract

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility on their input trained with a masked language modeling objective followed by multitask finetuning perform the best among our experiments. We therefore consider the adaptation of pretrained models across architectures and objectives. We find that pretrained non-causal decoder models can be adapted into performant generative causal decoder models, using autoregressive language modeling as a downstream task. Furthermore, we find that pretrained causal decoder models can be efficiently adapted into non-causal decoder models, ultimately achieving competitive performance after multitask finetuning. Code and checkpoints are available at https://github.com/bigscience-workshop/architecture-objective.