Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images

TL;DR

This paper introduces a deep learning model that estimates tactile properties like slipperiness and roughness from visual images, enabling robots to better understand and interact with their environment without manual labeling.

Contribution

It presents a novel encoder-decoder network that predicts tactile properties from images using only RGB and tactile data, without manual annotations.

Findings

Model generalizes to unseen materials

Learns meaningful visual-tactile feature associations

Achieves accurate tactile property estimation from images

Abstract

Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help us decide which actions we should choose and how to perform them. E.g., we can drive slower if we see that we have bad traction or grasp tighter if an item looks slippery. We believe that this ability also helps robots to enhance their understanding of the environment, and thus enables them to tailor their actions to the situation at hand. We therefore propose a model to estimate the degree of tactile properties from visual perception alone (e.g., the level of slipperiness or roughness). Our method extends a encoder-decoder network, in which the latent variables are visual and tactile features. In contrast to previous works, our method does not require manual labeling, but only RGB images and the corresponding tactile sensor data. All our data is collected with a webcam and uSkin tactile sensor mounted on the end-effector of a Sawyer robot, which strokes the surfaces of 25 different materials. We show that our model generalizes to materials not included in the training data by evaluating the feature space, indicating that it has learned to associate important tactile properties with images.