Domain-Specific Fine-Tuning and Prompt-Based Learning: A Comparative Study for developing Natural Language-Based BIM Information Retrieval Systems

TL;DR

This study compares domain-specific fine-tuning and prompt-based learning for natural language interfaces in BIM data retrieval, proposing a hybrid approach that enhances performance across intent recognition and question answering tasks.

Contribution

It introduces a systematic comparison of two NLP approaches for BIM information retrieval and proposes a hybrid method for improved system performance.

Findings

Fine-tuning excels in intent recognition accuracy.

Prompt-based learning, especially GPT-4o, performs well in table question answering.

Hybrid approach achieves balanced performance across tasks.

Abstract

Building Information Modeling (BIM) is essential for managing building data across the entire lifecycle, supporting tasks from design to maintenance. Natural Language Interface (NLI) systems are increasingly explored as user-friendly tools for information retrieval in Building Information Modeling (BIM) environments. Despite their potential, accurately extracting BIM-related data through natural language queries remains a persistent challenge due to the complexity use queries and specificity of domain knowledge. This study presents a comparative analysis of two prominent approaches for developing NLI-based BIM information retrieval systems: domain-specific fine-tuning and prompt-based learning using large language models (LLMs). A two-stage framework consisting of intent recognition and table-based question answering is implemented to evaluate the effectiveness of both approaches. To support this evaluation, a BIM-specific dataset of 1,740 annotated queries of varying types across 69 models is constructed. Experimental results show that domain-specific fine-tuning delivers superior performance in intent recognition tasks, while prompt-based learning, particularly with GPT-4o, shows strength in table-based question answering. Based on these findings, this study identify a hybrid configuration that combines fine-tuning for intent recognition with prompt-based learning for question answering, achieving more balanced and robust performance across tasks. This integrated approach is further tested through case studies involving BIM models of varying complexity. This study provides a systematic analysis of the strengths and limitations of each approach and discusses the applicability of the NLI to real-world BIM scenarios. The findings offer insights for researchers and practitioners in designing intelligent, language-driven BIM systems.