BERTian Poetics: Constrained Composition with Masked LMs

TL;DR

This paper explores how masked language models like BERT can be used for constrained text generation, inspired by OuLiPo poetics, by interpreting them as energy-based models suitable for sampling.

Contribution

It introduces a novel approach to constrained composition using masked language models interpreted as energy-based models, connecting NLP with poetic constraints.

Findings

Demonstrates the use of masked LMs for constrained text generation.

Links the methodology to OuLiPo poetics.

Provides insights into the poetics of generated text.

Abstract

Masked language models have recently been interpreted as energy-based sequence models that can be generated from using a Metropolis--Hastings sampler. This short paper demonstrates how this can be instrumentalized for constrained composition and explores the poetics implied by such a usage. Our focus on constraints makes it especially apt to understand the generated text through the poetics of the OuLiPo movement.