Sim-to-real for high-resolution optical tactile sensing: From images to 3D contact force distributions

TL;DR

This paper presents a simulation-based approach to generate synthetic tactile images for vision-based sensors, enabling accurate 3D contact force estimation from images without real-world data collection.

Contribution

The authors develop a finite element simulation method to generate synthetic tactile images and train neural networks for force estimation, eliminating the need for real-world training data.

Findings

High accuracy in real-world force estimation from synthetic data

Model transferability across different tactile sensors

Real-time inference capability

Abstract

The images captured by vision-based tactile sensors carry information about high-resolution tactile fields, such as the distribution of the contact forces applied to their soft sensing surface. However, extracting the information encoded in the images is challenging and often addressed with learning-based approaches, which generally require a large amount of training data. This article proposes a strategy to generate tactile images in simulation for a vision-based tactile sensor based on an internal camera that tracks the motion of spherical particles within a soft material. The deformation of the material is simulated in a finite element environment under a diverse set of contact conditions, and spherical particles are projected to a simulated image. Features extracted from the images are mapped to the 3D contact force distribution, with the ground truth also obtained via finite-element simulations, with an artificial neural network that is therefore entirely trained on synthetic data avoiding the need for real-world data collection. The resulting model exhibits high accuracy when evaluated on real-world tactile images, is transferable across multiple tactile sensors without further training, and is suitable for efficient real-time inference.