Uniform Complexity for Text Generation

TL;DR

This paper introduces UCTG, a benchmark for evaluating and improving the consistency of text complexity in generative NLP models, highlighting current models' struggles to maintain input prompt complexity.

Contribution

The paper presents a new benchmark, UCTG, and evaluates over 150 features to assess and enhance the uniformity of linguistic complexity in text generation.

Findings

Models like GPT-2 struggle to preserve input complexity.

Finetuning with professional texts does not fully solve complexity preservation.

The benchmark enables systematic evaluation of complexity consistency.

Abstract

Large language models (LLMs) have shown promising results in a wide array of generative NLP tasks, such as summarization and machine translation. In the context of narrative generation, however, existing models still do not capture factors that contribute to producing consistent text. For instance, it is logical that a piece of text or a story should be uniformly readable throughout and that this form of complexity should be controllable. As such, if the complexity of an input text prompt is rated first-grade reading level in the Flesch Reading Ease test, then the generated text continuing the plot should also be within this range of complexity. With this in mind, we introduce Uniform Complexity for Text Generation (UCTG), a new benchmark test which raises the challenge of making generative models observe uniform linguistic properties with respect to prompts. We experiment with over 150+ linguistically and cognitively motivated features for evaluating text complexity in humans and generative models. From our results, we find that models such as GPT-2 struggle to preserve the complexity of input prompts used in its generations, even if finetuned with professionally written texts.