GhostObjects: Instructing Robots by Manipulating Spatially Aligned Virtual Twins in Augmented Reality

TL;DR

This paper introduces GhostObjects, a novel AR interface allowing users to instruct robots by manipulating virtual twins of real objects, enabling precise and flexible task specification beyond traditional methods.

Contribution

It presents a new AR-based interaction paradigm using GhostObjects for robot instruction, enhancing precision and flexibility over existing programming methods.

Findings

Enables precise spatial manipulation of virtual object twins

Supports complex task specifications beyond pick-and-place

Improves human-robot interaction in AR environments

Abstract

Robots are increasingly capable of autonomous operations, yet human interaction remains essential for issuing personalized instructions. Instead of directly controlling robots through Programming by Demonstration (PbD) or teleoperation, we propose giving instructions by interacting with GhostObjects-world-aligned, life-size virtual twins of physical objects-in augmented reality (AR). By direct manipulation of GhostObjects, users can precisely specify physical goals and spatial parameters, with features including real-world lasso selection of multiple objects and snapping back to default positions, enabling tasks beyond simple pick-and-place.