Benchmarking and Evaluating VLMs for Software Architecture Diagram Understanding

TL;DR

This paper introduces SADU, a benchmark for evaluating vision-language models on understanding software architecture diagrams, revealing current models' limitations in diagram reasoning and grounding.

Contribution

The paper presents SADU, a new benchmark with curated diagrams and question-answer tasks, to assess and improve VLMs' understanding of software architecture diagrams.

Findings

Best model achieves only 70.18% accuracy on SADU tasks.

Current VLMs struggle with diagram reasoning and visual relation grounding.

Significant gap exists between current models and software engineering needs.

Abstract

Software architecture diagrams are important design artifacts for communicating system structure, behavior, and data organization throughout the software development lifecycle. Although recent progress in large language models has substantially advanced code-centric software engineering tasks such as code generation, testing, and maintenance, the ability of modern vision-language models (VLMs) to understand software architecture diagrams remains underexplored. To address this gap, we present SADU, a benchmark for Software Architecture Diagram Understanding that evaluates VLMs on architecture diagrams as structured software engineering artifacts rather than generic images. SADU contains 154 carefully curated diagrams spanning behavioral, structural, and ER diagrams, paired with structured annotations and 2,431 question-answer tasks covering counting and retrieval reasoning. We evaluate 11 state-of-the-art VLMs from the Gemini, Claude, GPT, and Qwen families. Our results show that software architecture diagram understanding remains challenging for current models: the best-performing model gemini-3-flash-preview achieves only 70.18\% accuracy, while gpt-4o-mini only achieves 17.77\% accuracy. The results further reveal the weaknesses in diagram reasoning and visual relation grounding, highlighting a gap between current VLMs and the needs of design-stage software engineering. SADU provides a foundation for future research on diagram-aware AI systems and more faithful AI-assisted software engineering workflows.