Seeing into Darkness: Scotopic Visual Recognition

TL;DR

This paper introduces a framework for low-photon image classification in the scotopic regime, enabling efficient object recognition with minimal photon usage while maintaining accuracy, applicable to fields like biomedical imaging and astronomy.

Contribution

It develops an adaptive, optimal speed-accuracy tradeoff algorithm for classifying objects directly from photon streams, outperforming traditional vision in low-light conditions.

Findings

Achieves comparable accuracy with less than 0.1% of photons used in conventional images.

Works effectively even with unknown and varying illumination levels.

Proposes a power-efficient hardware model using spiking neural networks and photon-counting sensors.

Abstract

Images are formed by counting how many photons traveling from a given set of directions hit an image sensor during a given time interval. When photons are few and far in between, the concept of `image' breaks down and it is best to consider directly the flow of photons. Computer vision in this regime, which we call `scotopic', is radically different from the classical image-based paradigm in that visual computations (classification, control, search) have to take place while the stream of photons is captured and decisions may be taken as soon as enough information is available. The scotopic regime is important for biomedical imaging, security, astronomy and many other fields. Here we develop a framework that allows a machine to classify objects with as few photons as possible, while maintaining the error rate below an acceptable threshold. A dynamic and asymptotically optimal speed-accuracy tradeoff is a key feature of this framework. We propose and study an algorithm to optimize the tradeoff of a convolutional network directly from lowlight images and evaluate on simulated images from standard datasets. Surprisingly, scotopic systems can achieve comparable classification performance as traditional vision systems while using less than 0.1% of the photons in a conventional image. In addition, we demonstrate that our algorithms work even when the illuminance of the environment is unknown and varying. Last, we outline a spiking neural network coupled with photon-counting sensors as a power-efficient hardware realization of scotopic algorithms.