LLM-Driven Augmented Reality Puppeteer: Controller-Free Voice-Commanded Robot Teleoperation

TL;DR

This paper presents a novel controller-free AR system that uses large language models and voice commands to enable intuitive, safe, and immersive robot teleoperation without physical controllers.

Contribution

It introduces a new AR puppeteering system driven by LLMs and NLP, eliminating the need for physical controllers in robot teleoperation.

Findings

System successfully enables controller-free robot control

User demonstration validates system functionality

Potential for safer, more intuitive robotic interaction

Abstract

The integration of robotics and augmented reality (AR) presents transformative opportunities for advancing human-robot interaction (HRI) by improving usability, intuitiveness, and accessibility. This work introduces a controller-free, LLM-driven voice-commanded AR puppeteering system, enabling users to teleoperate a robot by manipulating its virtual counterpart in real time. By leveraging natural language processing (NLP) and AR technologies, our system -- prototyped using Meta Quest 3 -- eliminates the need for physical controllers, enhancing ease of use while minimizing potential safety risks associated with direct robot operation. A preliminary user demonstration successfully validated the system's functionality, demonstrating its potential for safer, more intuitive, and immersive robotic control.