Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR

TL;DR

This paper introduces a modular AR agent system that combines large language models with grounded vision models to interpret complex natural language queries for spatial retrieval and scene understanding in augmented reality environments.

Contribution

It presents a novel modular architecture integrating MLLMs and perception tools for relational reasoning and spatial retrieval, supporting plug-and-play models without retraining.

Findings

Enables understanding of object relations and interactions in 3D space.

Supports complex language queries for spatial localization and reasoning.

Provides an evaluation framework for real-world spatial grounding tasks.

Abstract

Traditional augmented reality (AR) systems predominantly rely on fixed class detectors or fiducial markers, limiting their ability to interpret complex, open-vocabulary natural language queries. We present a modular AR agent system that integrates multimodal large language models (MLLMs) with grounded vision models to enable relational reasoning in space and language-conditioned spatial retrieval in physical environments. Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities, ranging from simple object identification to multi-object relational reasoning, while returning meter-accurate 3D anchors. It constructs dynamic AR scene graphs encoding nine typed relations (spatial, structural-semantic, causal-functional), enabling MLLMs to understand not just what objects exist, but how they relate and interact in 3D space. Through task-adaptive region-of-interest highlighting and contextual spatial retrieval, the system guides human attention to information-dense areas while supporting human-in-the-loop refinement. The agent dynamically invokes coordinate-aware tools for complex queries-selection, measurement, comparison, and actuation-grounding language understanding in physical operations. The modular architecture supports plug-and-use vision-language models without retraining, establishing AR agents as intermediaries that augment MLLMs with real-world spatial intelligence for interactive scene understanding. We also introduce GroundedAR-Bench, an evaluation framework for language-driven real world localization and relation grounding across diverse environments.