PixelRNN: In-pixel Recurrent Neural Networks for End-to-end-optimized Perception with Neural Sensors

TL;DR

PixelRNN is a novel in-sensor neural network architecture that encodes spatio-temporal features directly on the sensor using binary operations, significantly reducing data transmission for perception tasks on edge devices.

Contribution

It introduces PixelRNN, an efficient recurrent neural network design that leverages sensor programmability to perform in-sensor processing with minimal data transmission.

Findings

Reduces data transmission by 64x compared to traditional systems.

Achieves competitive accuracy in gesture recognition and lip reading.

Validated on SCAMP-5 sensor-processor platform.

Abstract

Conventional image sensors digitize high-resolution images at fast frame rates, producing a large amount of data that needs to be transmitted off the sensor for further processing. This is challenging for perception systems operating on edge devices, because communication is power inefficient and induces latency. Fueled by innovations in stacked image sensor fabrication, emerging sensor-processors offer programmability and minimal processing capabilities directly on the sensor. We exploit these capabilities by developing an efficient recurrent neural network architecture, PixelRNN, that encodes spatio-temporal features on the sensor using purely binary operations. PixelRNN reduces the amount of data to be transmitted off the sensor by a factor of 64x compared to conventional systems while offering competitive accuracy for hand gesture recognition and lip reading tasks. We experimentally validate PixelRNN using a prototype implementation on the SCAMP-5 sensor-processor platform.