Touch2Touch: Cross-Modal Tactile Generation for Object Manipulation

TL;DR

Touch2Touch introduces a cross-modal tactile generation method using diffusion models to translate signals between different sensors, enabling sensor-specific processing and improving object pose estimation in robotics.

Contribution

It presents a novel diffusion-based approach for cross-modal tactile signal translation between GelSlim and Soft Bubble sensors, facilitating sensor-agnostic tactile analysis.

Findings

Successful translation between GelSlim and Soft Bubble signals.

Improved in-hand object pose estimation using cross-modal signals.

Open-source dataset and code available for further research.

Abstract

Today's touch sensors come in many shapes and sizes. This has made it challenging to develop general-purpose touch processing methods since models are generally tied to one specific sensor design. We address this problem by performing cross-modal prediction between touch sensors: given the tactile signal from one sensor, we use a generative model to estimate how the same physical contact would be perceived by another sensor. This allows us to apply sensor-specific methods to the generated signal. We implement this idea by training a diffusion model to translate between the popular GelSlim and Soft Bubble sensors. As a downstream task, we perform in-hand object pose estimation using GelSlim sensors while using an algorithm that operates only on Soft Bubble signals. The dataset, the code, and additional details can be found at https://www.mmintlab.com/research/touch2touch/.