From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing

TL;DR

This paper introduces a new benchmark and framework for generating and editing structured diagrams directly from textual descriptions, addressing limitations of existing text-to-image and text-to-code methods.

Contribution

It presents DiagramGenBenchmark for evaluation and DiagramAgent, a modular framework for high-quality, editable diagram generation and refinement.

Findings

DiagramAgent outperforms baselines in accuracy and coherence

Extensive experiments validate the effectiveness of the framework

Benchmark covers eight diverse diagram categories

Abstract

We introduce the task of text-to-diagram generation, which focuses on creating structured visual representations directly from textual descriptions. Existing approaches in text-to-image and text-to-code generation lack the logical organization and flexibility needed to produce accurate, editable diagrams, often resulting in outputs that are either unstructured or difficult to modify. To address this gap, we introduce DiagramGenBenchmark, a comprehensive evaluation framework encompassing eight distinct diagram categories, including flowcharts, model architecture diagrams, and mind maps. Additionally, we present DiagramAgent, an innovative framework with four core modules-Plan Agent, Code Agent, Check Agent, and Diagram-to-Code Agent-designed to facilitate both the generation and refinement of complex diagrams. Our extensive experiments, which combine objective metrics with human evaluations, demonstrate that DiagramAgent significantly outperforms existing baseline models in terms of accuracy, structural coherence, and modifiability. This work not only establishes a foundational benchmark for the text-to-diagram generation task but also introduces a powerful toolset to advance research and applications in this emerging area.