AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding

TL;DR

This paper introduces AECV-Bench, a comprehensive benchmark to evaluate multimodal models on architectural and engineering drawings, revealing strengths in text extraction but weaknesses in spatial and symbol understanding.

Contribution

The paper presents AECV-Bench, a new benchmark for assessing multimodal models on AEC drawings, including object counting and document QA tasks, highlighting current model limitations.

Findings

High accuracy in OCR and text-based QA (up to 0.95)

Moderate performance in spatial reasoning tasks

Poor performance in symbol-centric counting (often 0.40-0.55 accuracy)

Abstract

AEC drawings encode geometry and semantics through symbols, layout conventions, and dense annotation, yet it remains unclear whether modern multimodal and vision-language models can reliably interpret this graphical language. We present AECV-Bench, a benchmark for evaluating multimodal and vision-language models on realistic AEC artefacts via two complementary use cases: (i) object counting on 120 high-quality floor plans (doors, windows, bedrooms, toilets), and (ii) drawing-grounded document QA spanning 192 question-answer pairs that test text extraction (OCR), instance counting, spatial reasoning, and comparative reasoning over common drawing regions. Object-counting performance is reported using per-field exact-match accuracy and MAPE results, while document-QA performance is reported using overall accuracy and per-category breakdowns with an LLM-as-a-judge scoring pipeline and targeted human adjudication for edge cases. Evaluating a broad set of state-of-the-art models under a unified protocol, we observe a stable capability gradient; OCR and text-centric document QA are strongest (up to 0.95 accuracy), spatial reasoning is moderate, and symbol-centric drawing understanding - especially reliable counting of doors and windows - remains unsolved (often 0.40-0.55 accuracy) with substantial proportional errors. These results suggest that current systems function well as document assistants but lack robust drawing literacy, motivating domain-specific representations and tool-augmented, human-in-the-loop workflows for an efficient AEC automation.