SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches

TL;DR

SketchAgent is a multi-agent system that automates converting hand-drawn sketches into structured, semantically coherent diagrams, reducing manual effort and enabling applications in design, education, and engineering.

Contribution

We introduce SketchAgent, a novel multi-agent framework that automates sketch-to-diagram conversion, and the Sketch2Diagram Benchmark for evaluating such systems.

Findings

SketchAgent effectively produces accurate diagrams from sketches.

The Sketch2Diagram Benchmark provides a comprehensive dataset for evaluation.

Our approach significantly reduces manual effort in diagram creation.

Abstract

Hand-drawn sketches are a natural and efficient medium for capturing and conveying ideas. Despite significant advancements in controllable natural image generation, translating freehand sketches into structured, machine-readable diagrams remains a labor-intensive and predominantly manual task. The primary challenge stems from the inherent ambiguity of sketches, which lack the structural constraints and semantic precision required for automated diagram generation. To address this challenge, we introduce SketchAgent, a multi-agent system designed to automate the transformation of hand-drawn sketches into structured diagrams. SketchAgent integrates sketch recognition, symbolic reasoning, and iterative validation to produce semantically coherent and structurally accurate diagrams, significantly reducing the need for manual effort. To evaluate the effectiveness of our approach, we propose the Sketch2Diagram Benchmark, a comprehensive dataset and evaluation framework encompassing eight diverse diagram categories, such as flowcharts, directed graphs, and model architectures. The dataset comprises over 6,000 high-quality examples with token-level annotations, standardized preprocessing, and rigorous quality control. By streamlining the diagram generation process, SketchAgent holds great promise for applications in design, education, and engineering, while offering a significant step toward bridging the gap between intuitive sketching and machine-readable diagram generation. The benchmark is released at https://huggingface.co/datasets/DiagramAgent/Sketch2Diagram-Benchmark.