Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

TL;DR

This paper investigates how scaling Transformer models affects downstream tasks, revealing that model shape and scaling protocols are crucial, and introduces more efficient models that reduce parameters and training time while maintaining performance.

Contribution

It demonstrates the importance of model shape in downstream fine-tuning, proposes improved scaling protocols, and releases pretrained models to advance research.

Findings

Model shape influences downstream fine-tuning performance.

Scaling protocols vary with compute regions.

Proposed models are 50% smaller and 40% faster with similar quality.

Abstract

There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.