An Embodied AR Navigation Agent: Integrating BIM with Retrieval-Augmented Generation for Language Guidance

TL;DR

This paper presents an embodied AR navigation system that combines BIM data with retrieval-augmented generation and multi-agent LLMs to provide flexible, natural language-guided navigation assistance with high usability.

Contribution

It introduces a novel integration of BIM with RAG and multi-agent LLMs for flexible, language-based AR navigation, enhancing interpretability and spatial reasoning capabilities.

Findings

Achieved a System Usability Scale score of 80.5 indicating excellent usability.

Demonstrated significant improvement in users' perception of system intelligence.

Validated system effectiveness through real-world user study.

Abstract

Delivering intelligent and adaptive navigation assistance in augmented reality (AR) requires more than visual cues, as it demands systems capable of interpreting flexible user intent and reasoning over both spatial and semantic context. Prior AR navigation systems often rely on rigid input schemes or predefined commands, which limit the utility of rich building data and hinder natural interaction. In this work, we propose an embodied AR navigation system that integrates Building Information Modeling (BIM) with a multi-agent retrieval-augmented generation (RAG) framework to support flexible, language-driven goal retrieval and route planning. The system orchestrates three language agents, Triage, Search, and Response, built on large language models (LLMs), which enables robust interpretation of open-ended queries and spatial reasoning using BIM data. Navigation guidance is delivered through an embodied AR agent, equipped with voice interaction and locomotion, to enhance user experience. A real-world user study yields a System Usability Scale (SUS) score of 80.5, indicating excellent usability, and comparative evaluations show that the embodied interface can significantly improves users' perception of system intelligence. These results underscore the importance and potential of language-grounded reasoning and embodiment in the design of user-centered AR navigation systems.