Augmenting Tactile Simulators with Real-like and Zero-Shot Capabilities

TL;DR

This paper introduces SightGAN, a bi-directional GAN that enhances tactile simulators with real-like and zero-shot capabilities, improving transfer from simulation to real-world tactile sensing for dexterous manipulation.

Contribution

SightGAN extends CycleGAN with additional loss components to accurately reconstruct tactile contact patterns, enabling zero-shot training for new sensors and bridging the reality gap in tactile simulation.

Findings

Generated images are real-like and preserve contact accuracy.

Zero-shot models trained on synthetic data perform well on real sensors.

The framework supports training reinforcement learning policies for manipulation.

Abstract

Simulating tactile perception could potentially leverage the learning capabilities of robotic systems in manipulation tasks. However, the reality gap of simulators for high-resolution tactile sensors remains large. Models trained on simulated data often fail in zero-shot inference and require fine-tuning with real data. In addition, work on high-resolution sensors commonly focus on ones with flat surfaces while 3D round sensors are essential for dexterous manipulation. In this paper, we propose a bi-directional Generative Adversarial Network (GAN) termed SightGAN. SightGAN relies on the early CycleGAN while including two additional loss components aimed to accurately reconstruct background and contact patterns including small contact traces. The proposed SightGAN learns real-to-sim and sim-to-real processes over difference images. It is shown to generate real-like synthetic images while maintaining accurate contact positioning. The generated images can be used to train zero-shot models for newly fabricated sensors. Consequently, the resulted sim-to-real generator could be built on top of the tactile simulator to provide a real-world framework. Potentially, the framework can be used to train, for instance, reinforcement learning policies of manipulation tasks. The proposed model is verified in extensive experiments with test data collected from real sensors and also shown to maintain embedded force information within the tactile images.