Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

TL;DR

This paper investigates the variability in fine-tuning pretrained language models, revealing how random seed choices affect performance, and offers practical guidelines to improve stability and reproducibility in NLP tasks.

Contribution

It systematically analyzes the impact of weight initialization and data order on fine-tuning variability, providing new insights and best practices for NLP model training.

Findings

Performance varies significantly with random seed choices.

Some weight initializations are consistently effective across tasks.

Early stopping can prevent divergence on small datasets.

Abstract

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.