Conversational Interfaces for Parametric Conceptual Architectural Design: Integrating Mixed Reality with LLM-driven Interaction

TL;DR

This paper presents a novel mixed reality interface that combines speech, gestures, and large language models to facilitate intuitive parametric architectural design within immersive environments, overcoming previous limitations.

Contribution

It introduces a conversational MR system integrating LLMs for dynamic parametric modeling, enabling non-programmers to manipulate complex geometries intuitively.

Findings

Reduces cognitive barriers in early-stage design

Enables real-time parametric model manipulation

Supports natural embodied interaction in MR environments

Abstract

Mixed reality (MR) environments offer embodied spatial interaction, providing intuitive 3D manipulation capabilities that enhance the conceptual design process. Parametric modeling, a powerful and advanced architectural design method, enables the generation of complex, optimized geometries. However, its integration into MR environments remains limited due to precision constraints and unsuitable input modalities. Existing MR tools prioritize spatial interaction but lack the control and expressiveness required for parametric workflows, particularly for designers without formal programming backgrounds. We address this gap by introducing a novel conversational MR interface that combines speech input, gesture recognition, and a multi-agent large language model (LLM) system to support intuitive parametric modeling. Our system dynamically manages parameter states, resolves ambiguous commands through conversation and contextual prompting, and enables real-time model manipulation within immersive environments. We demonstrate how this approach reduces cognitive and operational barriers in early-stage design tasks, allowing users to refine and explore their design space. This work expands the role of MR to a generative design platform, supporting programmatic thinking in design tasks through natural, embodied interaction.