From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration

TL;DR

This paper introduces VisPainter, a multi-agent framework for creating editable, high-density scientific illustrations with element-level control, and proposes VisBench, a comprehensive benchmark for evaluating such illustrations.

Contribution

The paper presents a novel multi-agent system for scientific illustration that enables editable, component-wise control and introduces a new benchmark for evaluation.

Findings

VisPainter enables true element-level editing of scientific diagrams.

VisBench provides a multi-dimensional assessment of illustration quality.

Vision-language models are evaluated, revealing their strengths and limitations.

Abstract

Scientific illustrations demand both high information density and post-editability. However, current generative models have two major limitations: Frist, image generation models output rasterized images lacking semantic structure, making it impossible to access, edit, or rearrange independent visual components in the images. Second, code-based generation methods (TikZ or SVG), although providing element-level control, force users into the cumbersome cycle of "writing-compiling-reviewing" and lack the intuitiveness of manipulation. Neither of these two approaches can well meet the needs for efficiency, intuitiveness, and iterative modification in scientific creation. To bridge this gap, we introduce VisPainter, a multi-agent framework for scientific illustration built upon the model context protocol. VisPainter orchestrates three specialized modules-a Manager, a Designer, and a Toolbox-to collaboratively produce diagrams compatible with standard vector graphics software. This modular, role-based design allows each element to be explicitly represented and manipulated, enabling true element-level control and any element can be added and modified later. To systematically evaluate the quality of scientific illustrations, we introduce VisBench, a benchmark with seven-dimensional evaluation metrics. It assesses high-information-density scientific illustrations from four aspects: content, layout, visual perception, and interaction cost. To this end, we conducted extensive ablation experiments to verify the rationality of our architecture and the reliability of our evaluation methods. Finally, we evaluated various vision-language models, presenting fair and credible model rankings along with detailed comparisons of their respective capabilities. Additionally, we isolated and quantified the impacts of role division, step control,and description on the quality of illustrations.