Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation

TL;DR

This paper demonstrates that tactile sensing can be used to enable camera-free robotic telemanipulation by reconstructing 3D objects in VR and providing haptic feedback, especially useful in conditions where cameras fail.

Contribution

It introduces a novel approach combining tactile-based 3D object reconstruction with haptic feedback for camera-free teleoperation in robotic manipulation.

Findings

Successful pick-and-place tasks without cameras

Effective tactile-based 3D object reconstruction in VR

Potential for complex applications in challenging environments

Abstract

Dexterous robotic manipulator teleoperation is widely used in many applications, either where it is convenient to keep the human inside the control loop, or to train advanced robot agents. So far, this technology has been used in combination with camera systems with remarkable success. On the other hand, only a limited number of studies have focused on leveraging haptic feedback from tactile sensors in contexts where camera-based systems fail, such as due to self-occlusions or poor light conditions like smoke. This study demonstrates the feasibility of precise pick-and-place teleoperation without cameras by leveraging tactile-based 3D object reconstruction in VR and providing haptic feedback to a blindfolded user. Our preliminary results show that integrating these technologies enables the successful completion of telemanipulation tasks previously dependent on cameras, paving the way for more complex future applications.