Deep Tactile Experience: Estimating Tactile Sensor Output from Depth Sensor Data

TL;DR

This paper introduces a neural network-based method to estimate tactile sensor outputs from depth data alone, enabling non-invasive tactile sensing for robots by leveraging surface images and past interactions.

Contribution

It presents a novel approach to predict tactile responses from depth data using a neural network trained on a specialized dataset, reducing the need for physical contact in tactile sensing.

Findings

Achieved SSIM score of 0.84, outperforming the baseline.

Demonstrated accurate tactile estimation from depth images.

Validated the method's effectiveness with statistical significance.

Abstract

Tactile sensing is inherently contact based. To use tactile data, robots need to make contact with the surface of an object. This is inefficient in applications where an agent needs to make a decision between multiple alternatives that depend the physical properties of the contact location. We propose a method to get tactile data in a non-invasive manner. The proposed method estimates the output of a tactile sensor from the depth data of the surface of the object based on past experiences. An experience dataset is built by allowing the robot to interact with various objects, collecting tactile data and the corresponding object surface depth data. We use the experience dataset to train a neural network to estimate the tactile output from depth data alone. We use GelSight tactile sensors, an image-based sensor, to generate images that capture detailed surface features at the contact location. We train a network with a dataset containing 578 tactile-image to depthmap correspondences. Given a depth-map of the surface of an object, the network outputs an estimate of the response of the tactile sensor, should it make a contact with the object. We evaluate the method with structural similarity index matrix (SSIM), a similarity metric between two images commonly used in image processing community. We present experimental results that show the proposed method outperforms a baseline that uses random images with statistical significance getting an SSIM score of 0.84 +/- 0.0056 and 0.80 +/- 0.0036, respectively.