Ultra-low-light computer vision using trained photon correlations

TL;DR

This paper introduces correlation-aware training (CAT), a novel method that optimizes correlated-photon illumination and a Transformer-based classifier to improve object recognition accuracy in ultra-low-light, noisy conditions.

Contribution

The work presents a new end-to-end training approach that leverages photon correlations for enhanced low-light object recognition, surpassing traditional image reconstruction methods.

Findings

Achieved up to 15 percentage points improvement in classification accuracy.

Demonstrated effectiveness with fewer than 100 photon shots.

Showed superiority over uncorrelated and untrained correlated illumination.

Abstract

Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene -- such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical-electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number (<= 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task -- object recognition -- and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction.