SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

TL;DR

SciFlow-Bench is a new benchmark for evaluating scientific diagram generation models based on their ability to produce structurally correct diagrams from pixel images, emphasizing the importance of structural recoverability over visual similarity.

Contribution

It introduces SciFlow-Bench, a structure-first evaluation framework that pairs real scientific PDFs with ground-truth graphs and assesses models through inverse parsing in a closed-loop system.

Findings

Structural correctness is challenging, especially for complex diagrams.

Current models struggle with preserving structural information.

The benchmark highlights the need for structure-aware diagram generation.

Abstract

Scientific diagrams convey explicit structural information, yet modern text-to-image models often produce visually plausible but structurally incorrect results. Existing benchmarks either rely on image-centric or subjective metrics insensitive to structure, or evaluate intermediate symbolic representations rather than final rendered images, leaving pixel-based diagram generation underexplored. We introduce SciFlow-Bench, a structure-first benchmark for evaluating scientific diagram generation directly from pixel-level outputs. Built from real scientific PDFs, SciFlow-Bench pairs each source framework figure with a canonical ground-truth graph and evaluates models as black-box image generators under a closed-loop, round-trip protocol that inverse-parses generated diagram images back into structured graphs for comparison. This design enforces evaluation by structural recoverability rather than visual similarity alone, and is enabled by a hierarchical multi-agent system that coordinates planning, perception, and structural reasoning. Experiments show that preserving structural correctness remains a fundamental challenge, particularly for diagrams with complex topology, underscoring the need for structure-aware evaluation.