Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

TL;DR

This paper introduces Ishigaki-IDS-Bench, a benchmark for evaluating how well large language models can generate industry-standard XML-based Information Delivery Specifications from BIM requirements, highlighting current limitations and supporting future development.

Contribution

It provides a new benchmark dataset and evaluation framework for assessing LLMs' ability to generate compliant IDS XML from BIM data, including a comprehensive set of expert-verified examples and evaluation tools.

Findings

Best LLM achieves 65.6% macro F1 in content agreement.

Only 27.7% of outputs pass the Content audit.

Current LLMs struggle to reliably generate standard-compliant IDS XML.

Abstract

Large language models (LLMs) are widely used to generate structured outputs such as JSON, SQL, and code, yet public resources remain limited for evaluating generation that must simultaneously satisfy industry-standard XML and domain vocabulary constraints. This paper presents Ishigaki-IDS-Bench, a benchmark for evaluating the ability to generate Information Delivery Specification (IDS) XML from Building Information Modeling (BIM) information requirements. The benchmark contains 166 BIM/IDS expert-authored and verified examples created by expanding 83 practical scenarios into Japanese and English, corresponding gold IDS files, and metadata for input format, language, turn setting, IFC version, and construction domain. Its evaluation combines IDSAuditTool-based Processability, Structure, and Content audits with content-agreement evaluation against gold IDS files. In zero-shot evaluation over 10 LLMs, the best model reaches 65.6% macro F1 for content agreement, while only 27.7% of outputs pass the Content audit. These results show that current LLMs can express part of the information requirements as IDS, but still struggle to stably generate XML that satisfies the IDS standard and IFC vocabulary constraints. Ishigaki-IDS-Bench supports comparative evaluation, failure analysis, and the development of constrained structured generation methods that conform to domain standards. We release the evaluation scripts and benchmark data under the CC BY 4.0 license on GitHub and Hugging Face.