Event-Driven Visual-Tactile Sensing and Learning for Robots

TL;DR

This paper introduces an event-driven visual-tactile perception system with a novel tactile sensor and spike-based learning, enabling fast, power-efficient robot perception for tasks like classification and slip detection.

Contribution

It presents a biologically-inspired tactile sensor and a multi-modal spike-based neural network for efficient, fast perception in robots, with publicly available datasets for further research.

Findings

Achieved good accuracy on container classification and slip detection tasks.

Demonstrated scalability and efficiency of the neuromorphic tactile sensor.

Provided datasets to facilitate future multi-modal event-driven perception research.

Abstract

This work contributes an event-driven visual-tactile perception system, comprising a novel biologically-inspired tactile sensor and multi-modal spike-based learning. Our neuromorphic fingertip tactile sensor, NeuTouch, scales well with the number of taxels thanks to its event-based nature. Likewise, our Visual-Tactile Spiking Neural Network (VT-SNN) enables fast perception when coupled with event sensors. We evaluate our visual-tactile system (using the NeuTouch and Prophesee event camera) on two robot tasks: container classification and rotational slip detection. On both tasks, we observe good accuracies relative to standard deep learning methods. We have made our visual-tactile datasets freely-available to encourage research on multi-modal event-driven robot perception, which we believe is a promising approach towards intelligent power-efficient robot systems.