Understanding and Mitigating Network Latency Effect on Teleoperated-Robot with Extended Reality

TL;DR

This paper introduces TeleXR, an open-source XR teleoperation framework that reduces network latency effects on remote robot control by local sensing, adaptive scaling, and contention-aware scheduling, improving accuracy and robustness.

Contribution

The paper presents TeleXR, a novel end-to-end XR teleoperation system that decouples control and visualization from network issues, enabling more reliable remote robot operation.

Findings

Significantly reduces teleoperation errors caused by network latency.

Maintains high robot planning accuracy despite network degradation.

Enhances robustness with bandwidth-adaptive and contention-aware features.

Abstract

Robot teleoperation with extended reality (XR teleoperation) enables intuitive interaction by allowing remote robots to mimic user motions with real-time 3D feedback. However, existing systems face significant motion-to-motion (M2M) latency--the delay between the user's latest motion and the corresponding robot feedback--leading to high teleoperation error and mission completion time. This issue stems from the system's exclusive reliance on network communication, making it highly vulnerable to network degradation. To address these challenges, we introduce TeleXR, the first end-to-end, fully open-sourced XR teleoperation framework that decouples robot control and XR visualization from network dependencies. TeleXR leverages local sensing data to reconstruct delayed or missing information of the counterpart, thereby significantly reducing network-induced issues. This approach allows both the XR and robot to run concurrently with network transmission while maintaining high robot planning accuracy. TeleXR also features contention-aware scheduling to mitigate GPU contention and bandwidth-adaptive point cloud scaling to cope with limited bandwidth.