BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

TL;DR

This paper demonstrates that BERT can be modeled as a Markov random field language model, enabling sentence sampling that produces diverse yet slightly less fluent sentences compared to traditional models.

Contribution

It introduces a novel formulation of BERT as a Markov random field, allowing natural sentence sampling and analysis of its generative capabilities.

Findings

BERT can be effectively sampled as a Markov random field.

Generated sentences are more diverse than traditional models.

Sentence quality is slightly lower but still high.

Abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.