CEQuest: Benchmarking Large Language Models for Construction Estimation

TL;DR

This paper introduces CEQuest, a benchmark dataset for evaluating large language models in construction estimation tasks, revealing current models' limitations and emphasizing the need for domain-specific knowledge integration.

Contribution

The paper presents CEQuest, the first specialized benchmark dataset for construction estimation, and evaluates five state-of-the-art LLMs, highlighting their performance gaps in construction-related tasks.

Findings

Current LLMs show significant room for improvement in construction tasks.

Performance varies across models in accuracy, execution time, and size.

Open-sourcing CEQuest will support future domain-specific model development.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of general-domain tasks. However, their effectiveness in specialized fields, such as construction, remains underexplored. In this paper, we introduce CEQuest, a novel benchmark dataset specifically designed to evaluate the performance of LLMs in answering construction-related questions, particularly in the areas of construction drawing interpretation and estimation. We conduct comprehensive experiments using five state-of-the-art LLMs, including Gemma 3, Phi4, LLaVA, Llama 3.3, and GPT-4.1, and evaluate their performance in terms of accuracy, execution time, and model size. Our experimental results demonstrate that current LLMs exhibit considerable room for improvement, highlighting the importance of integrating domain-specific knowledge into these models. To facilitate further research, we will open-source the proposed CEQuest dataset, aiming to foster the development of specialized large language models (LLMs) tailored to the construction domain.