Multimodal Proximity and Visuotactile Sensing With a Selectively Transmissive Soft Membrane

TL;DR

This paper introduces a novel multimodal sensor combining visuotactile and proximity sensing using a soft membrane, enabling improved robotic manipulation during object contact and motion transitions.

Contribution

The work presents a new sensor design integrating RGB, air pressure, and ToF cameras with a transmissive membrane for simultaneous visuotactile and proximity sensing.

Findings

Sensor effectively captures tactile and proximity data

Demonstrated in robotic approaching, catching, and throwing tasks

Fusion of modalities enhances dynamic manipulation capabilities

Abstract

The most common sensing modalities found in a robot perception system are vision and touch, which together can provide global and highly localized data for manipulation. However, these sensing modalities often fail to adequately capture the behavior of target objects during the critical moments as they transition out of static, controlled contact with an end-effector to dynamic and uncontrolled motion. In this work, we present a novel multimodal visuotactile sensor that provides simultaneous visuotactile and proximity depth data. The sensor integrates an RGB camera and air pressure sensor to sense touch with an infrared time-of-flight (ToF) camera to sense proximity by leveraging a selectively transmissive soft membrane to enable the dual sensing modalities. We present the mechanical design, fabrication techniques, algorithm implementations, and evaluation of the sensor's tactile and proximity modalities. The sensor is demonstrated in three open-loop robotic tasks: approaching and contacting an object, catching, and throwing. The fusion of tactile and proximity data could be used to capture key information about a target object's transition behavior for sensor-based control in dynamic manipulation.