PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing

TL;DR

PaperOrchestra is a multi-agent framework that automates the transformation of raw research materials into comprehensive, submission-ready AI research papers, including literature review and visual content, outperforming existing autonomous writers.

Contribution

It introduces a flexible multi-agent system for automated paper writing and a new benchmark with evaluators to assess its performance against baselines.

Findings

Outperforms autonomous baselines in literature review quality by 50%-68%.

Achieves 14%-38% higher overall manuscript quality.

Uses PaperWritingBench, a benchmark based on 200 top-tier AI papers.

Abstract

Synthesizing unstructured research materials into manuscripts is an essential yet under-explored challenge in AI-driven scientific discovery. Existing autonomous writers are rigidly coupled to specific experimental pipelines, and produce superficial literature reviews. We introduce PaperOrchestra, a multi-agent framework for automated AI research paper writing. It flexibly transforms unconstrained pre-writing materials into submission-ready LaTeX manuscripts, including comprehensive literature synthesis and generated visuals, such as plots and conceptual diagrams. To evaluate performance, we present PaperWritingBench, the first standardized benchmark of reverse-engineered raw materials from 200 top-tier AI conference papers, alongside a comprehensive suite of automated evaluators. In side-by-side human evaluations, PaperOrchestra significantly outperforms autonomous baselines, achieving an absolute win rate margin of 50%-68% in literature review quality, and 14%-38% in overall manuscript quality.