GelSplitter: Tactile Reconstruction from Near Infrared and Visible Images

TL;DR

GelSplitter introduces a multi-modal tactile sensing framework using synchronized RGB and NIR cameras, enabling improved 3D tactile reconstruction with a compact design and flexible sensor configurations.

Contribution

This work presents a novel multi-modal VT sensor framework with synchronized cameras and a neural network for enhanced tactile surface reconstruction.

Findings

RGB and NIR fusion improves reconstruction accuracy.

The framework supports various camera combinations, including thermal imaging.

The sensor maintains a compact size comparable to existing VT sensors.

Abstract

The GelSight-like visual tactile (VT) sensor has gained popularity as a high-resolution tactile sensing technology for robots, capable of measuring touch geometry using a single RGB camera. However, the development of multi-modal perception for VT sensors remains a challenge, limited by the mono camera. In this paper, we propose the GelSplitter, a new framework approach the multi-modal VT sensor with synchronized multi-modal cameras and resemble a more human-like tactile receptor. Furthermore, we focus on 3D tactile reconstruction and implement a compact sensor structure that maintains a comparable size to state-of-the-art VT sensors, even with the addition of a prism and a near infrared (NIR) camera. We also design a photometric fusion stereo neural network (PFSNN), which estimates surface normals of objects and reconstructs touch geometry from both infrared and visible images. Our results demonstrate that the accuracy of RGB and NIR fusion is higher than that of RGB images alone. Additionally, our GelSplitter framework allows for a flexible configuration of different camera sensor combinations, such as RGB and thermal imaging.