Should We Still Pretrain Encoders with Masked Language Modeling?

TL;DR

This paper investigates whether Masked Language Modeling (MLM) or Causal Language Modeling (CLM) is more effective for pretraining encoders, finding that a combined biphasic approach offers optimal performance and efficiency.

Contribution

It provides a comprehensive large-scale analysis comparing MLM and CLM pretraining, introducing a biphasic training strategy that leverages both objectives for improved results.

Findings

MLM generally outperforms in text representation tasks.

CLM models are more data-efficient and stable during fine-tuning.

Sequential CLM then MLM training yields the best performance within fixed compute budgets.

Abstract

Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked Language Modeling (MLM), recent evidence suggests that decoder models pretrained with Causal Language Modeling (CLM) can be effectively repurposed as encoders, often surpassing traditional encoders on text representation benchmarks. However, it remains unclear whether these gains reflect an inherent advantage of the CLM objective or arise from confounding factors such as model and data scale. In this paper, we address this question through a series of large-scale, carefully controlled pretraining ablations, training a total of 38 models ranging from 210 million to 1 billion parameters, and conducting over 15,000 fine-tuning and evaluation runs. We find that while training with MLM generally yields better performance across text representation tasks, CLM-trained models are more data-efficient and demonstrate improved fine-tuning stability. Building on these findings, we experimentally show that a biphasic training strategy that sequentially applies CLM and then MLM, achieves optimal performance under a fixed computational training budget. Moreover, we demonstrate that this strategy becomes more appealing when initializing from readily available pretrained CLM models, reducing the computational burden needed to train best-in-class encoder models. We release all project artifacts at https://hf.co/MLMvsCLM to foster further research.