GPT or BERT: why not both?

TL;DR

This paper introduces GPT-BERT, a hybrid transformer model combining masked and causal language modeling, demonstrating improved pretraining performance and flexibility for various NLP tasks.

Contribution

It presents a novel hybrid training objective that merges masked and causal language modeling within a single transformer, enabling versatile use and better performance.

Findings

Hybrid pretraining outperforms masked-only models

Hybrid pretraining outperforms causal-only models

Models and code are openly released

Abstract

We present a simple way to merge masked language modeling with causal language modeling. This hybrid training objective results in a model that combines the strengths of both modeling paradigms within a single transformer stack: GPT-BERT can be transparently used like any standard causal or masked language model. We test the pretraining process that enables this flexible behavior on the BabyLM Challenge 2024. The results show that the hybrid pretraining outperforms masked-only or causal-only models. We openly release the models, training corpora and code.