Qwen-BIM: developing large language model for BIM-based design with domain-specific benchmark and dataset

TL;DR

This paper introduces Qwen-BIM, a domain-specific large language model for BIM-based design, supported by a new benchmark and dataset, demonstrating significant performance improvements over general LLMs in construction-related tasks.

Contribution

The paper develops the first domain-specific LLM for BIM-based design, along with a comprehensive benchmark and dataset for evaluation and fine-tuning.

Findings

Qwen-BIM achieves a 21.0% increase in G-Eval score.

Performance of Qwen-BIM with 14B parameters rivals larger general LLMs.

The benchmark effectively assesses LLM capabilities in BIM tasks.

Abstract

As the construction industry advances toward digital transformation, BIM (Building Information Modeling)-based design has become a key driver supporting intelligent construction. Despite Large Language Models (LLMs) have shown potential in promoting BIM-based design, the lack of specific datasets and LLM evaluation benchmarks has significantly hindered the performance of LLMs. Therefore, this paper addresses this gap by proposing: 1) an evaluation benchmark for BIM-based design together with corresponding quantitative indicators to evaluate the performance of LLMs, 2) a method for generating textual data from BIM and constructing corresponding BIM-derived datasets for LLM evaluation and fine-tuning, and 3) a fine-tuning strategy to adapt LLMs for BIM-based design. Results demonstrate that the proposed domain-specific benchmark effectively and comprehensively assesses LLM capabilities, highlighting that general LLMs are still incompetent for domain-specific tasks. Meanwhile, with the proposed benchmark and datasets, Qwen-BIM is developed and achieves a 21.0% average increase in G-Eval score compared to the base LLM model. Notably, with only 14B parameters, performance of Qwen-BIM is comparable to that of general LLMs with 671B parameters for BIM-based design tasks. Overall, this study develops the first domain-specific LLM for BIM-based design by introducing a comprehensive benchmark and high-quality dataset, which provide a solid foundation for developing BIM-related LLMs in various fields.