Immersive Human-in-the-Loop Control: Real-Time 3D Surface Meshing and Physics Simulation

TL;DR

This paper presents TactiMesh, a real-time 3D surface meshing and physics simulation system for immersive human-in-the-loop robot control, enhancing teleoperation with predictive visual and haptic feedback via HMD.

Contribution

The novel TactiMesh system integrates SLAM, space carving, and physics simulation to provide real-time 3D environment modeling and haptic feedback for improved teleoperation.

Findings

Enhanced manipulation speed in remote tasks

Reduced delay in haptic feedback response

Improved immersion and control accuracy

Abstract

This paper introduces the TactiMesh Teleoperator Interface (TTI), a novel predictive visual and haptic system designed explicitly for human-in-the-loop robot control using a head-mounted display (HMD). By employing simultaneous localization and mapping (SLAM)in tandem with a space carving method (CARV), TTI creates a real time 3D surface mesh of remote environments from an RGB camera mounted on a Barrett WAM arm. The generated mesh is integrated into a physics simulator, featuring a digital twin of the WAM robot arm to create a virtual environment. In this virtual environment, TTI provides haptic feedback directly in response to the operator's movements, eliminating the problem with delayed response from the haptic follower robot. Furthermore, texturing the 3D mesh with keyframes from SLAM allows the operator to control the viewpoint of their Head Mounted Display (HMD) independently of the arm-mounted robot camera, giving a better visual immersion and improving manipulation speed. Incorporating predictive visual and haptic feedback significantly improves teleoperation in applications such as search and rescue, inspection, and remote maintenance.