5G Virtual Reality Manipulator Teleoperation using a Mobile Phone

TL;DR

This paper introduces a 5G-enabled VR teleoperation system where a mobile phone controls a manipulator using IMU and camera data, providing immersive visual and haptic feedback for precise remote manipulation.

Contribution

It presents a novel mobile phone-based teleoperation approach integrating VR visual feedback and haptic force feedback over wireless networks.

Findings

Effective Cartesian pose estimation using phone sensors

High immersion through multi-view VR visualization

Real-time force feedback via phone haptic actuator

Abstract

This paper presents an approach to teleoperate a manipulator using a mobile phone as a leader device. Using its IMU and camera, the phone estimates its Cartesian pose which is then used to to control the Cartesian pose of the robot's tool. The user receives visual feedback in the form of multi-view video - a point cloud rendered in a virtual reality environment. This enables the user to observe the scene from any position. To increase immersion, the robot's estimate of external forces is relayed using the phone's haptic actuator. Leader and follower are connected through wireless networks such as 5G or Wi-Fi. The paper describes the setup and analyzes its performance.