Pretrained Language Model Embryology: The Birth of ALBERT

TL;DR

This paper studies the developmental process of ALBERT during pretraining, revealing how linguistic and world knowledge evolve and highlighting that more pretraining does not always lead to better knowledge or performance.

Contribution

It introduces the concept of 'embryology' of pretrained language models and analyzes the knowledge acquisition process during ALBERT's pretraining.

Findings

ALBERT learns to predict different parts of speech at different speeds.

Knowledge and performance do not necessarily improve with more pretraining.

Pretraining stages show varying levels of linguistic and world knowledge.

Abstract

While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process from a set of randomly initialized parameters to a totipotent language model, which we refer to as the embryology of a pretrained language model. Our results show that ALBERT learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks' performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We will provide source codes and pretrained models to reproduce our results at https://github.com/d223302/albert-embryology.