Marker or Markerless? Mode-Switchable Optical Tactile Sensing for Diverse Robot Tasks

TL;DR

This paper introduces a mode-switchable optical tactile sensor that can seamlessly transition between marker and markerless modes, enhancing robot perception and manipulation capabilities without extra hardware.

Contribution

It proposes a novel approach using generative and regressive models to enable smooth mode transitions in optical tactile sensors, addressing a key limitation of uni-modal sensors.

Findings

Reduces misclassification in perception tasks by 2 categories

Improves contact area segmentation IoU by 3.53%

Achieves a 92.59% success rate in slip detection for manipulation

Abstract

Optical tactile sensors play a pivotal role in robot perception and manipulation tasks. The membrane of these sensors can be painted with markers or remain markerless, enabling them to function in either marker or markerless mode. However, this uni-modal selection means the sensor is only suitable for either manipulation or perception tasks. While markers are vital for manipulation, they can also obstruct the camera, thereby impeding perception. The dilemma of selecting between marker and markerless modes presents a significant obstacle. To address this issue, we propose a novel mode-switchable optical tactile sensing approach that facilitates transitions between the two modes. The marker-to-markerless transition is achieved through a generative model, whereas its inverse transition is realized using a sparsely supervised regressive model. Our approach allows a single-mode optical sensor to operate effectively in both marker and markerless modes without the need for additional hardware, making it well-suited for both perception and manipulation tasks. Extensive experiments validate the effectiveness of our method. For perception tasks, our approach decreases the number of categories that include misclassified samples by 2 and improves contact area segmentation IoU by 3.53%. For manipulation tasks, our method attains a high success rate of 92.59% in slip detection. Code, dataset and demo videos are available at the project website: https://gitouni.github.io/Marker-Markerless-Transition/