Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Papers

TL;DR

This paper introduces a large-scale benchmark for automated diagram generation from scientific papers and proposes Paper2SysArch, an end-to-end system that effectively converts papers into structured diagrams, advancing research in scientific visualization.

Contribution

It establishes the first comprehensive benchmark with 3,000 papers and ground-truth diagrams, and presents Paper2SysArch, a novel system leveraging multi-agent collaboration for diagram generation.

Findings

Achieved a composite score of 69.0 on complex papers.

Created the first benchmark with 3,000 paper-diagram pairs.

Demonstrated the effectiveness of multi-agent collaboration in diagram generation.

Abstract

The manual creation of system architecture diagrams for scientific papers is a time-consuming and subjective process, while existing generative models lack the necessary structural control and semantic understanding for this task. A primary obstacle hindering research and development in this domain has been the profound lack of a standardized benchmark to quantitatively evaluate the automated generation of diagrams from text. To address this critical gap, we introduce a novel and comprehensive benchmark, the first of its kind, designed to catalyze progress in automated scientific visualization. It consists of 3,000 research papers paired with their corresponding high-quality ground-truth diagrams and is accompanied by a three-tiered evaluation metric assessing semantic accuracy, layout coherence, and visual quality. Furthermore, to establish a strong baseline on this new benchmark, we propose Paper2SysArch, an end-to-end system that leverages multi-agent collaboration to convert papers into structured, editable diagrams. To validate its performance on complex cases, the system was evaluated on a manually curated and more challenging subset of these papers, where it achieves a composite score of 69.0. This work's principal contribution is the establishment of a large-scale, foundational benchmark to enable reproducible research and fair comparison. Meanwhile, our proposed system serves as a viable proof-of-concept, demonstrating a promising path forward for this complex task.