From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality

TL;DR

Speech-to-Spatial is a framework that converts spoken instructions into spatially grounded AR guidance, improving remote assistance by reducing guidance iterations and cognitive load.

Contribution

It introduces a speech-only referent disambiguation system grounded in an object-centric relational graph for AR guidance, without relying on gestures or manual annotations.

Findings

Improves task efficiency in remote guidance scenarios.

Reduces cognitive load compared to voice-only systems.

Enhances usability with visually explainable AR guidance.

Abstract

We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.