Long range teleoperation for fine manipulation tasks under time-delay network conditions

TL;DR

This paper introduces a semi-autonomous teleoperation system optimized for long-range, high-latency networks that uses vision guidance and a coarse-to-fine control approach, enabling precise manipulation without prior scene knowledge.

Contribution

It presents a novel coarse-to-fine teleoperation framework that combines self-exploration, 3D reconstruction, and image-based visual servoing for effective remote manipulation under time delays.

Findings

Successfully operated over long distances from Singapore to Canada.

Reduced network data transmission by sending sparse points and images.

Effective in performing multiple manipulation tasks in high latency conditions.

Abstract

We present a coarse-to-fine approach based semi-autonomous teleoperation system using vision guidance. The system is optimized for long range teleoperation tasks under time-delay network conditions and does not require prior knowledge of the remote scene. Our system initializes with a self exploration behavior that senses the remote surroundings through a freely mounted eye-in-hand web cam. The self exploration stage estimates hand-eye calibration and provides a telepresence interface via real-time 3D geometric reconstruction. The human operator is able to specify a visual task through the interface and a coarse-to-fine controller guides the remote robot enabling our system to work in high latency networks. Large motions are guided by coarse 3D estimation, whereas fine motions use image cues (IBVS). Network data transmission cost is minimized by sending only sparse points and a final image to the human side. Experiments from Singapore to Canada on multiple tasks were conducted to show our system's capability to work in long range teleoperation tasks.