Neuromorphic visual attention for Sign-language recognition on SpiNNaker

TL;DR

This paper presents a neuromorphic system for real-time, energy-efficient American Sign Language recognition using event-driven visual attention and spiking neural networks on SpiNNaker hardware.

Contribution

It introduces an end-to-end neuromorphic architecture combining visual attention and a compact spiking neural network for sign language recognition on neuromorphic hardware.

Findings

Achieved 92.27% accuracy in simulation.

Deployed on SpiNNaker with 83.1% accuracy.

Energy consumption of 0.565 mW and 3 ms latency.

Abstract

Sign-language recognition has achieved substantial gains in classification accuracy in recent years; however, the latency and power requirements of most existing methods limit their suitability for real-time deployment. Neuromorphic sensing and processing offer an alternative paradigm based on sparse, event-driven computation that supports low-latency and energy-efficient perception. In this work, we introduce an end-to-end neuromorphic architecture for American Sign Language (ASL) fingerspelling recognition that integrates a spiking visual attention mechanism for online region-of-interest extraction with a compact spiking neural network deployed on the SpiNNaker neuromorphic platform. We benchmark the proposed system against two datasets: a synthetically generated event-based version of the Sign Language MNIST dataset and a natively recorded ASL-DVS dataset, whilst providing a comprehensive overview of Sign-language recognition and related work. This work yields competitive performance in simulation (92.27%) and comparable performance on neuromorphic hardware deployment (83.1%), while achieving the most energy-efficient architecture (0.565 mW) and low latency (3 ms) across all benchmarked approaches. Despite its compact design, the system demonstrates the suitability of task-dependent visual attention applications for edge deployment.