Seeing Through your Skin: Recognizing Objects with a Novel Visuotactile Sensor

TL;DR

This paper introduces the See-Through-your-Skin sensor (STS), a novel visuotactile device that combines visual and tactile sensing in a single hardware, enabling improved object recognition and property inference.

Contribution

The paper presents a new semitransparent sensor capable of dual visual and tactile sensing, along with a deep learning fusion architecture for multi-modal object recognition.

Findings

Enhanced object classification accuracy with multi-modal data

Successful recognition of fine textures and physical properties

Effective validation through simulations and real-world experiments

Abstract

We introduce a new class of vision-based sensor and associated algorithmic processes that combine visual imaging with high-resolution tactile sending, all in a uniform hardware and computational architecture. We demonstrate the sensor's efficacy for both multi-modal object recognition and metrology. Object recognition is typically formulated as an unimodal task, but by combining two sensor modalities we show that we can achieve several significant performance improvements. This sensor, named the See-Through-your-Skin sensor (STS), is designed to provide rich multi-modal sensing of contact surfaces. Inspired by recent developments in optical tactile sensing technology, we address a key missing feature of these sensors: the ability to capture a visual perspective of the region beyond the contact surface. Whereas optical tactile sensors are typically opaque, we present a sensor with a semitransparent skin that has the dual capabilities of acting as a tactile sensor and/or as a visual camera depending on its internal lighting conditions. This paper details the design of the sensor, showcases its dual sensing capabilities, and presents a deep learning architecture that fuses vision and touch. We validate the ability of the sensor to classify household objects, recognize fine textures, and infer their physical properties both through numerical simulations and experiments with a smart countertop prototype.