Real-time image processing with a 2D semiconductor neural network vision sensor

TL;DR

This paper introduces a novel 2D semiconductor neural network vision sensor that can simultaneously sense and process images in real-time, reducing data redundancy and power consumption in machine vision systems.

Contribution

The authors present a reconfigurable 2D semiconductor photodiode array that functions as an integrated neural network, enabling real-time image classification and encoding without latency.

Findings

Achieved a processing throughput of 20 million bins per second.

Demonstrated both supervised and unsupervised learning capabilities.

Successfully trained the sensor to classify and encode images optically projected onto it.

Abstract

In recent years, machine vision has taken huge leaps and is now becoming an integral part of various intelligent systems, including autonomous vehicles, robotics, and many others. Usually, visual information is captured by a frame-based camera, converted into a digital format, and processed afterwards using a machine learning algorithm such as an artificial neural network (ANN). A large amount of (mostly redundant) data being passed through the entire signal chain, however, results in low frame rates and large power consumption. Various visual data preprocessing techniques have thus been developed that allow to increase the efficiency of the subsequent signal processing in an ANN. Here, we demonstrate that an image sensor itself can constitute an ANN that is able to simultaneously sense and process optical images without latency. Our device is based on a reconfigurable two-dimensional (2D) semiconductor photodiode array, with the synaptic weights of the network being stored in a continuously tunable photoresponsivity matrix. We demonstrate both supervised and unsupervised learning and successfully train the sensor to classify and encode images, that are optically projected onto the chip, with a throughput of 20 million bins per second.