BIMgent: Towards Autonomous Building Modeling via Computer-use Agents

TL;DR

BIMgent is a novel agentic framework utilizing multimodal large language models to automate and improve the efficiency of building modeling tasks in BIM software, demonstrating significant success over baselines.

Contribution

This work introduces BIMgent, the first autonomous agent framework for BIM modeling that integrates multimodal LLMs to perform GUI operations in architectural design.

Findings

Achieved a 32% success rate in real-world modeling tasks.

Outperformed baseline models which failed to complete tasks.

Reduced manual workload while maintaining design quality.

Abstract

Existing computer-use agents primarily focus on general-purpose desktop automation tasks, with limited exploration of their application in highly specialized domains. In particular, the 3D building modeling process in the Architecture, Engineering, and Construction (AEC) sector involves open-ended design tasks and complex interaction patterns within Building Information Modeling (BIM) authoring software, which has yet to be thoroughly addressed by current studies. In this paper, we propose BIMgent, an agentic framework powered by multimodal large language models (LLMs), designed to enable autonomous building model authoring via graphical user interface (GUI) operations. BIMgent automates the architectural building modeling process, including multimodal input for conceptual design, planning of software-specific workflows, and efficient execution of the authoring GUI actions. We evaluate BIMgent on real-world building modeling tasks, including both text-based conceptual design generation and reconstruction from existing building design. The design quality achieved by BIMgent was found to be reasonable. Its operations achieved a 32% success rate, whereas all baseline models failed to complete the tasks (0% success rate). Results demonstrate that BIMgent effectively reduces manual workload while preserving design intent, highlighting its potential for practical deployment in real-world architectural modeling scenarios. Project page: https://tumcms.github.io/BIMgent.github.io/