SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

TL;DR

SuperWriter introduces an agent-based framework with structured planning and refinement stages, significantly improving long-form text coherence and quality in large language models through hierarchical optimization and human-like thinking processes.

Contribution

It presents a novel framework combining structured thinking, hierarchical preference optimization, and fine-tuning to enhance long-form generation in LLMs, outperforming larger models.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Outperforms larger baseline models in automatic and human evaluations.

Ablation studies confirm the effectiveness of hierarchical DPO and structured thinking.

Abstract

Long-form text generation remains a significant challenge for large language models (LLMs), particularly in maintaining coherence, ensuring logical consistency, and preserving text quality as sequence length increases. To address these limitations, we propose SuperWriter-Agent, an agent-based framework designed to enhance the quality and consistency of long-form text generation. SuperWriter-Agent introduces explicit structured thinking-through planning and refinement stages into the generation pipeline, guiding the model to follow a more deliberate and cognitively grounded process akin to that of a professional writer. Based on this framework, we construct a supervised fine-tuning dataset to train a 7B SuperWriter-LM. We further develop a hierarchical Direct Preference Optimization (DPO) procedure that uses Monte Carlo Tree Search (MCTS) to propagate final quality assessments and optimize each generation step accordingly. Empirical results across diverse benchmarks demonstrate that SuperWriter-LM achieves state-of-the-art performance, surpassing even larger-scale baseline models in both automatic evaluation and human evaluation. Furthermore, comprehensive ablation studies demonstrate the effectiveness of hierarchical DPO and underscore the value of incorporating structured thinking steps to improve the quality of long-form text generation.