Gesture First, LLM-Assisted Voice Complement: Exploring Multimodal Robot 'Puppeteer' Teleoperation Via Virtual Counterpart in Augmented Reality

TL;DR

This study compares gesture-only and voice+gesture interaction modes in AR-based robot teleoperation, revealing trade-offs between efficiency, flexibility, and user workload, and offers design guidelines for multimodal control.

Contribution

It introduces a multimodal AR 'puppeteer' system for robot control and empirically evaluates the impact of gesture and voice modalities on performance and user experience.

Findings

Gesture-only control is more reliable and efficient for time-critical tasks.

Voice+gesture control offers greater flexibility but can increase workload due to latency.

Prior robotics expertise influences user performance and experience.

Abstract

Robot teleoperation via augmented reality (AR) offers a promising path toward more intuitive human-robot interaction (HRI). We present a head-mounted AR 'puppeteer' system in which users control a physical robot by interacting with its virtual counterpart robot using large language model (LLM)-assisted voice commands and hand-gesture interaction on the Meta Quest 3. In a within-subject user study with 42 participants performing an AR-based robotic pick-and-place pattern-matching task, we empirically compare two interaction conditions: gesture-only (GO) and combined voice+gesture (VG) on performance and user experience (UX). In VG, voice and gesture operate in a sequential role-allocated manner, with voice handling high-level navigation and gesture handling fine manipulation. Our results show that GO currently provides more reliable and efficient control for this time-critical task, while VG introduces additional flexibility but also latency and recognition issues that can increase workload. We additionally analyze how prior robotics expertise differentiates performance and UX across conditions. Based on these findings, we distill a set of design guidelines for AR 'puppeteer' metaphoric robot teleoperation, framing multimodality as an adaptive strategy that must balance efficiency, robustness, and user expertise rather than assuming that additional modalities are universally beneficial.