AntLM: Bridging Causal and Masked Language Models

TL;DR

AntLM introduces a new language modeling paradigm that combines causal and masked training objectives, improving training efficiency and performance of foundation models by leveraging the strengths of both approaches.

Contribution

The paper proposes AntLM, a novel paradigm integrating CLM and MLM, demonstrating improved training performance on foundation models.

Findings

AntLM improves macro-average performance by 1% on BabyLlama.

AntLM achieves a 2.2% increase over baselines on LTG-BERT.

Combining CLM and MLM leverages their respective advantages.

Abstract

Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each paradigm in downstream tasks have shown a mix of advantages and disadvantages. In the past BabyLM Challenge 2023, although the MLM paradigm achieved the best average performance, the CLM paradigm demonstrated significantly faster convergence rates. For the BabyLM Challenge 2024, we propose a novel language modeling paradigm named $\textbf{AntLM}$, which integrates both CLM and MLM to leverage the advantages of these two classic paradigms. We chose the strict-small track and conducted experiments on two foundation models: BabyLlama, representing CLM, and LTG-BERT, representing MLM. During the training process for specific foundation models, we alternate between applying CLM or MLM training objectives and causal or bidirectional attention masks. Experimental results show that combining the two pretraining objectives leverages their strengths, enhancing overall training performance. Under the same epochs, $AntLM_{BabyLlama}$ improves Macro-average by 1%, and $AntLM_{LTG-BERT}$ achieves a 2.2% increase over the baselines.