BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
Zihan Deng, Changyu Du, Stavros Nousias, Andr\'e Borrmann

TL;DR
BIMgent is a novel agentic framework utilizing multimodal large language models to automate and improve the efficiency of building modeling tasks in BIM software, demonstrating significant success over baselines.
Contribution
This work introduces BIMgent, the first autonomous agent framework for BIM modeling that integrates multimodal LLMs to perform GUI operations in architectural design.
Findings
Achieved a 32% success rate in real-world modeling tasks.
Outperformed baseline models which failed to complete tasks.
Reduced manual workload while maintaining design quality.
Abstract
Existing computer-use agents primarily focus on general-purpose desktop automation tasks, with limited exploration of their application in highly specialized domains. In particular, the 3D building modeling process in the Architecture, Engineering, and Construction (AEC) sector involves open-ended design tasks and complex interaction patterns within Building Information Modeling (BIM) authoring software, which has yet to be thoroughly addressed by current studies. In this paper, we propose BIMgent, an agentic framework powered by multimodal large language models (LLMs), designed to enable autonomous building model authoring via graphical user interface (GUI) operations. BIMgent automates the architectural building modeling process, including multimodal input for conceptual design, planning of software-specific workflows, and efficient execution of the authoring GUI actions. We evaluate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBIM and Construction Integration · Innovations in Concrete and Construction Materials · Multimodal Machine Learning Applications
