Toward a Predictive eXtended Reality Teleoperation System with Duo-Virtual Spaces

TL;DR

This paper proposes a novel duo-virtual spaces design to optimize latency in XR teleoperation systems, enabling more precise and rapid robot control by localizing agents and objects on the user side and calibrating with ground-truth data.

Contribution

It introduces a duo-virtual spaces architecture and a calibration method to reduce latency and improve control accuracy in XR teleoperation systems.

Findings

Profiling reveals significant latency issues in current XR teleoperation.

Duo-virtual spaces reduce end-to-end latency.

Calibration improves positional accuracy.

Abstract

Extended Reality (XR) provides a more intuitive interaction method for teleoperating robots compared to traditional 2D controls. Recent studies have laid the groundwork for usable teleoperation with XR, but it fails in tasks requiring rapid motion and precise manipulations due to the large delay between user motion and agent feedback. In this work, we profile the end-to-end latency in a state-of-the-art XR teleoperation system and propose our idea to optimize the latency by implementing a duo-virtual spaces design and localizing the agent and objects in the user-side virtual space, while calibrating with periodic ground-truth poses from the agent-side virtual space.