Reconfigurable, large-format D-ToF/photon-counting SPAD image sensors with embedded FPGA for scene adaptability

TL;DR

This paper presents a reconfigurable large-format SPAD image sensor with embedded FPGA processing at the pixel level, enabling efficient photon counting, timestamp processing, and neural network reprogramming for adaptable imaging applications.

Contribution

It introduces an on-chip FPGA architecture integrated with SPADs for real-time processing and reconfigurability, enhancing scene adaptability and power efficiency.

Findings

Hierarchical processing reduces power consumption.

Embedded FPGA enables reprogramming of neural networks.

Effective photon counting and timestamp processing achieved.

Abstract

CMOS-compatible single-photon avalanche diodes (SPADs) have emerged in many systems as the solution of choice for cameras with photon-number resolution and photon counting capabilities. Being natively digital optical interfaces, SPADs are naturally drawn to in situ logic processing and event-driven computation; they are usually coupled to discrete FPGAs to enable reconfigurability. In this work, we propose to bring the FPGA on-chip, in direct contact with the SPADs at pixel or cluster level. To demonstrate the suitability of this approach, we created an architecture for processing timestamps and photon counts using programmable weighted sums based on an efficient use of look-up tables. The outputs are processed hierarchically, similarly to what is done in FPGAs, reducing power consumption and simplifying I/Os. Finally, we show how artificial neural networks can be designed and reprogrammed by using look-up tables in an efficient way.