From Soft Materials to Controllers with NeuroTouch: A Neuromorphic Tactile Sensor for Real-Time Gesture Recognition

TL;DR

NeuroTouch is an optical tactile sensor combining soft materials and neuromorphic vision for real-time gesture recognition, achieving high accuracy and low localization error, with potential applications in AR/VR and gaming.

Contribution

This work introduces NeuroTouch, a novel neuromorphic tactile sensor that integrates soft materials with event-based vision for gesture detection, and provides a new benchmark dataset.

Findings

Over 91% gesture classification accuracy

3.41 mm finger localization error

0.96 mm gesture intensity error

Abstract

This work presents NeuroTouch, an optical-based tactile sensor that combines a highly deformable dome-shaped soft material with an integrated neuromorphic camera, leveraging frame-based and dynamic vision for gesture detection. Our approach transforms an elastic body into a rich and nuanced interactive controller by tracking markers printed on its surface with event-based methods and harnessing their trajectories through RANSAC-based techniques. To benchmark our framework, we have created a 25 min gesture dataset, which we make publicly available to foster research in this area. Achieving over 91% accuracy in gesture classification, a 3.41 mm finger localization distance error, and a 0.96 mm gesture intensity error, our real-time, lightweight, and low-latency pipeline holds promise for applications in video games, augmented/virtual reality, and accessible devices. This research lays the groundwork for advancements in gesture detection for vision-based soft-material input technologies. Dataset: Coming Soon, Video: Coming Soon