SridBench: Benchmark of Scientific Research Illustration Drawing of Image Generation Model

TL;DR

SridBench is a new benchmark designed to evaluate AI models' ability to generate accurate and clear scientific illustrations, revealing current models still lag behind human performance in this complex task.

Contribution

This paper introduces SridBench, the first comprehensive benchmark for scientific figure generation, addressing a critical gap in evaluating AI's capabilities in technical visual creation.

Findings

Top models like GPT-4o-image underperform humans in scientific figure generation.

Models struggle with text clarity, visual quality, and scientific accuracy.

Benchmark highlights the need for more advanced reasoning in AI visual generation.

Abstract

Recent years have seen rapid advances in AI-driven image generation. Early diffusion models emphasized perceptual quality, while newer multimodal models like GPT-4o-image integrate high-level reasoning, improving semantic understanding and structural composition. Scientific illustration generation exemplifies this evolution: unlike general image synthesis, it demands accurate interpretation of technical content and transformation of abstract ideas into clear, standardized visuals. This task is significantly more knowledge-intensive and laborious, often requiring hours of manual work and specialized tools. Automating it in a controllable, intelligent manner would provide substantial practical value. Yet, no benchmark currently exists to evaluate AI on this front. To fill this gap, we introduce SridBench, the first benchmark for scientific figure generation. It comprises 1,120 instances curated from leading scientific papers across 13 natural and computer science disciplines, collected via human experts and MLLMs. Each sample is evaluated along six dimensions, including semantic fidelity and structural accuracy. Experimental results reveal that even top-tier models like GPT-4o-image lag behind human performance, with common issues in text/visual clarity and scientific correctness. These findings highlight the need for more advanced reasoning-driven visual generation capabilities.