3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception

TL;DR

This paper presents GelSplitter3D, a novel 3D vision-tactile reconstruction method using infrared and visible images, achieving improved accuracy for robotic tactile perception on biomimetic curved surfaces.

Contribution

It introduces a new imaging system, a neural network for normal estimation, and a boundary-aware surface integration method for better tactile sensing on curved surfaces.

Findings

40% improvement in normal estimation accuracy

Enhanced tactile sensing performance in grasping tasks

Effective reconstruction on biomimetic curved surfaces

Abstract

To achieve human-like haptic perception in anthropomorphic grippers, the compliant sensing surfaces of vision tactile sensor (VTS) must evolve from conventional planar configurations to biomimetically curved topographies with continuous surface gradients. However, planar VTSs have challenges when extended to curved surfaces, including insufficient lighting of surfaces, blurring in reconstruction, and complex spatial boundary conditions for surface structures. With an end goal of constructing a human-like fingertip, our research (i) develops GelSplitter3D by expanding imaging channels with a prism and a near-infrared (NIR) camera, (ii) proposes a photometric stereo neural network with a CAD-based normal ground truth generation method to calibrate tactile geometry, and (iii) devises a normal integration method with boundary constraints of depth prior information to correcting the cumulative error of surface integrals. We demonstrate better tactile sensing performance, a 40$\%$ improvement in normal estimation accuracy, and the benefits of sensor shapes in grasping and manipulation tasks.