Integrating Planning into Single-Turn Long-Form Text Generation

TL;DR

This paper introduces a planning-based auxiliary task for large language models to improve the quality of long-form text generation, demonstrating significant enhancements in coherence and relevance across multiple datasets.

Contribution

It presents a novel single auxiliary task that enables LLMs to plan and structure content without multiple prompting rounds, utilizing synthetic data to improve long-form text generation.

Findings

+2.5% ROUGE-Lsum improvement

3.60 win/loss ratio in human evaluations

Enhanced organization, relevance, and verifiability

Abstract

Generating high-quality, in-depth textual documents, such as academic papers, news articles, Wikipedia entries, and books, remains a significant challenge for Large Language Models (LLMs). In this paper, we propose to use planning to generate long form content. To achieve our goal, we generate intermediate steps via an auxiliary task that teaches the LLM to plan, reason and structure before generating the final text. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning. To overcome the scarcity of training data for these intermediate steps, we leverage LLMs to generate synthetic intermediate writing data such as outlines, key information and summaries from existing full articles. Our experiments demonstrate on two datasets from different domains, namely the scientific news dataset SciNews and Wikipedia datasets in KILT-Wiki and FreshWiki, that LLMs fine-tuned with the auxiliary task generate higher quality documents. We observed +2.5% improvement in ROUGE-Lsum, and a strong 3.60 overall win/loss ratio via human SxS evaluation, with clear wins in organization, relevance, and verifiability.