Future Sight: Dynamic Story Generation with Large Pretrained Language Models

TL;DR

Future Sight introduces a method for fine-tuning pretrained transformers to condition story generation on future plot points, enabling more controllable and coherent narratives guided by human input.

Contribution

The paper presents a novel fine-tuning approach allowing transformers to attend to future plot events, enhancing controllability in story generation tasks.

Findings

Improves coherence when conditioning on future plot points

Enables human-guided narrative steering during generation

Achieves positive evaluations from human judges

Abstract

Recent advances in deep learning research, such as transformers, have bolstered the ability for automated agents to generate creative texts similar to those that a human would write. By default, transformer decoders can only generate new text with respect to previously generated text. The output distribution of candidate tokens at any position is conditioned on previously selected tokens using a self-attention mechanism to emulate the property of autoregression. This is inherently limiting for tasks such as controllable story generation where it may be necessary to condition on future plot events when writing a story. In this work, we propose Future Sight, a method for finetuning a pretrained generative transformer on the task of future conditioning. Transformer decoders are typically pretrained on the task of completing a context, one token at a time, by means of self-attention. Future Sight additionally enables a decoder to attend to an encoded future plot event. This motivates the decoder to expand on the context in a way that logically concludes with the provided future. During inference, the future plot event can be written by a human author to steer the narrative being generated in a certain direction. We evaluate the efficacy of our approach on a story generation task with human evaluators.