Machine vision with small numbers of detected photons per inference

TL;DR

This paper introduces photon-aware neuromorphic sensing (PANS), a novel end-to-end optimized approach for machine vision in extremely low-light conditions, achieving high accuracy with very few detected photons per inference.

Contribution

The paper presents PANS, a new method that incorporates photon detection statistics into training, enabling effective machine vision in photon-starved scenarios, which was previously very challenging.

Findings

Achieved 73% accuracy on FashionMNIST with only 4.9 photons per inference.

Achieved 86% accuracy on MNIST with only 8.6 photons per inference.

Demonstrated potential for applications in quantum and other photon-starved sensing setups.

Abstract

Machine vision, including object recognition and image reconstruction, is a central technology in many consumer devices and scientific instruments. The design of machine-vision systems has been revolutionized by the adoption of end-to-end optimization, in which the optical front end and the post-processing back end are jointly optimized. However, while machine vision currently works extremely well in moderate-light or bright-light situations -- where a camera may detect thousands of photons per pixel and billions of photons per frame -- it is far more challenging in very low-light situations. We introduce photon-aware neuromorphic sensing (PANS), an approach for end-to-end optimization in highly photon-starved scenarios. The training incorporates knowledge of the low photon budget and the stochastic nature of light detection when the average number of photons per pixel is near or less than 1. We report a proof-of-principle experimental demonstration in which we performed low-light image classification using PANS, achieving 73% (82%) accuracy on FashionMNIST with an average of only 4.9 (17) detected photons in total per inference, and 86% (97%) on MNIST with 8.6 (29) detected photons -- orders of magnitude more photon-efficient than conventional approaches. We also report simulation studies showing how PANS could be applied to other classification, event-detection, and image-reconstruction tasks. By taking into account the statistics of measurement results for non-classical states or alternative sensing hardware, PANS could in principle be adapted to enable high-accuracy results in quantum and other photon-starved setups.