Facts2Story: Controlling Text Generation by Key Facts

TL;DR

This paper introduces a controlled text generation task that expands given facts into narratives, proposes new evaluation metrics, and compares methods, highlighting the strengths and weaknesses of different models in adhering to facts while maintaining fluency.

Contribution

It presents a new fact-based controlled generation task, human evaluation metrics, a dataset creation method, and a comparative analysis of fine-tuned models for fact adherence and fluency.

Findings

GPT2 produces more fluent text but less fact adherence.

XLNet-based plan-and-cloze model balances fluency and fact adherence.

New evaluation metrics for fact-controlled text generation.

Abstract

Recent advancements in self-attention neural network architectures have raised the bar for open-ended text generation. Yet, while current methods are capable of producing a coherent text which is several hundred words long, attaining control over the content that is being generated -- as well as evaluating it -- are still open questions. We propose a controlled generation task which is based on expanding a sequence of facts, expressed in natural language, into a longer narrative. We introduce human-based evaluation metrics for this task, as well as a method for deriving a large training dataset. We evaluate three methods on this task, based on fine-tuning pre-trained models. We show that while auto-regressive, unidirectional Language Models such as GPT2 produce better fluency, they struggle to adhere to the requested facts. We propose a plan-and-cloze model (using fine-tuned XLNet) which produces competitive fluency while adhering to the requested content.