DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning

TL;DR

DiagrammerGPT is a two-stage framework that leverages large language models to generate accurate, complex diagrams with detailed object layouts and labels, surpassing existing text-to-image models.

Contribution

The paper introduces a novel two-stage diagram generation approach using LLMs for planning and a dedicated diagram generator, along with a new benchmark dataset for evaluation.

Findings

Outperforms existing T2I models in diagram accuracy

Enables open-domain and multi-platform diagram generation

Supports human-in-the-loop editing and multimodal planning

Abstract

Text-to-image (T2I) generation has seen significant growth over the past few years. Despite this, there has been little work on generating diagrams with T2I models. A diagram is a symbolic/schematic representation that explains information using structurally rich and spatially complex visualizations (e.g., a dense combination of related objects, text labels, directional arrows/lines, etc.). Existing state-of-the-art T2I models often fail at diagram generation because they lack fine-grained object layout control when many objects are densely connected via complex relations such as arrows/lines, and also often fail to render comprehensible text labels. To address this gap, we present DiagrammerGPT, a novel two-stage text-to-diagram generation framework leveraging the layout guidance capabilities of LLMs to generate more accurate diagrams. In the first stage, we use LLMs to generate and iteratively refine 'diagram plans' (in a planner-auditor feedback loop). In the second stage, we use a diagram generator, DiagramGLIGEN, and a text label rendering module to generate diagrams (with clear text labels) following the diagram plans. To benchmark the text-to-diagram generation task, we introduce AI2D-Caption, a densely annotated diagram dataset built on top of the AI2D dataset. We show that our DiagrammerGPT framework produces more accurate diagrams, outperforming existing T2I models. We also provide comprehensive analysis, including open-domain diagram generation, multi-platform vector graphic diagram generation, human-in-the-loop editing, and multimodal planner/auditor LLMs.