BERTian Poetics: Constrained Composition with Masked LMs
Christopher Akiki, Martin Potthast

TL;DR
This paper explores how masked language models like BERT can be used for constrained text generation, inspired by OuLiPo poetics, by interpreting them as energy-based models suitable for sampling.
Contribution
It introduces a novel approach to constrained composition using masked language models interpreted as energy-based models, connecting NLP with poetic constraints.
Findings
Demonstrates the use of masked LMs for constrained text generation.
Links the methodology to OuLiPo poetics.
Provides insights into the poetics of generated text.
Abstract
Masked language models have recently been interpreted as energy-based sequence models that can be generated from using a Metropolis--Hastings sampler. This short paper demonstrates how this can be instrumentalized for constrained composition and explores the poetics implied by such a usage. Our focus on constraints makes it especially apt to understand the generated text through the poetics of the OuLiPo movement.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Music and Audio Processing
