Megapixel Photon-Counting Color Imaging using Quanta Image Sensor

TL;DR

This paper introduces the first megapixel color Quanta Image Sensor capable of high-resolution, low-light photon counting, achieved through a novel joint demosaicing-denoising algorithm that effectively reconstructs color images from extremely sparse photon data.

Contribution

The paper presents a pioneering megapixel color QIS with a specialized reconstruction algorithm, enabling high-resolution color imaging in extremely low-light conditions.

Findings

Successful demonstration of 1024x1024 color QIS imaging

Effective reconstruction at a mean exposure of a few photons per pixel

Superior low-light color imaging performance

Abstract

Quanta Image Sensor (QIS) is a single-photon detector designed for extremely low light imaging conditions. Majority of the existing QIS prototypes are monochrome based on single-photon avalanche diodes (SPAD). Passive color imaging has not been demonstrated with single-photon detectors due to the intrinsic difficulty of shrinking the pixel size and increasing the spatial resolution while maintaining acceptable intra-pixel cross-talk. In this paper, we present image reconstruction of the first color QIS with a resolution of $1024 \times 1024$ pixels, supporting both single-bit and multi-bit photon counting capability. Our color image reconstruction is enabled by a customized joint demosaicing-denoising algorithm, leveraging truncated Poisson statistics and variance stabilizing transforms. Experimental results of the new sensor and algorithm demonstrate superior color imaging performance for very low-light conditions with a mean exposure of as low as a few photons per pixel in both real and simulated images.

Figures

Equations

$$\mathbf{v}=[{\mathbf{v}}_r;{\mathbf{v}}_g;{\mathbf{v}}_b]=\alpha \mathbf{G}\mathbf{c}\in {\mathbb{R}}^{3M},$$

$$\mathbf{\theta }={\mathbf{S}}_r{\mathbf{v}}_r+{\mathbf{S}}_g{\mathbf{v}}_g+{\mathbf{S}}_b{\mathbf{v}}_b\in {\mathbb{R}}^M.$$

$$\mathbf{\theta }=\alpha \mathbf{S}\mathbf{G}\mathbf{c}.$$

$$\mathbb{P}(\mathbf{Y}=\mathbf{y})=\prod _{m=1}^M\frac{{\theta }_m^{y_m}{e}^{-{\theta }_m}}{y_m!}.$$

$$B_m=\{\begin{array}{ll}{Y}_m,& Y_m<2^L-1,\\ 2^L-1,& Y_m\ge 2^L-1.\end{array}$$

$$\mathbb{P}(\mathbf{Y}=\mathbf{y})=\prod _{t=1}^T\prod _{m=1}^M\frac{{\theta }_m^{y_{m,t}}{e}^{-{\theta }_m}}{y_{m,t}!},$$

$$Z_m=\sum _{t=1}^T{B}_{m,t}.$$

$$\mathbb{P}[Z_m=k|{\theta }_m]=(\genfrac{}{}{0ex}{}{T}{k}){\Psi }_q({\theta }_m{)}^{1-k}(1-{\Psi }_q({\theta }_m){)}^k,$$

$$\mathbb{P}[Z_m=k|{\theta }_m]=\frac{(T{\theta }_m{)}^k{e}^{-T{\theta }_m}}{k!}.$$

$$\mathbf{\theta }$$

$$\stackrel{\text{def}}{=}\mathcal{M}(\mathbf{z})=\{\begin{array}{ll}{\Psi }_q^{-1}(1-\mathbf{z}/T),& \text{mbox}Single-bit,\\ \mathbf{z}/T,& \text{mbox}Multi-bit.\end{array}$$

$$\mathbf{\beta }=\mathcal{T}(\mathbf{z})\stackrel{\text{def}}{=}\{\begin{array}{ll}\sqrt{T+\frac{1}{2}}{\sin }^{-1}\sqrt{\frac{\mathbf{z}+\frac{3}{8}}{T+\frac{3}{4}}},& \text{mbox}Single-bit,\\ \sqrt{\mathbf{z}+\frac{3}{8}},& \text{mbox}Multi-bit.\end{array}$$

$$\mathcal{T}(\mathbf{z})=\mathcal{T}(\mathbf{S}\tilde{\mathbf{z}})=\mathbf{S}\mathcal{T}(\tilde{\mathbf{z}}).$$

$$\hat{v}=\underset{\mathbf{x}}{\text{mbox}argmin}\Vert \mathbf{S}\mathbf{x}-\mathcal{T}(\mathbf{z}){\Vert }^2+\lambda g(\mathbf{x}),$$

$${\mathbf{x}}^{(k+1)}=({\mathbf{S}}^T\mathbf{S}+\rho \mathbf{I}{)}^{-1}({\mathbf{S}}^T\mathcal{T}(\mathbf{z})+\rho ({\mathbf{v}}^{(k)}-{\mathbf{u}}^{(k)})),$$

$${\mathbf{v}}^{(k+1)}={\mathcal{D}}_{\rho /\lambda }({\mathbf{x}}^{(k+1)}+{\mathbf{u}}^{(k)}),$$

Full text

††journal: oe††articletype: Research Article

\homepage

https://engineering.purdue.edu/ChanGroup/

1 Introduction

Quanta image sensor (QIS) is a concept proposed by E. Fossum in 2005 aiming to overcome the limited sensor performance with the continuously shrinking pixel size [1, 2]. In a CMOS image sensor [3], images are acquired by accumulating hundreds to thousands of photoelectrons in each pixel and converting the charges to voltages and then to digital signals. As pixel size shrinks, the amount of photons that fall on individual pixels drops. Under low-light (or high-speed imaging) conditions, photon shot noise dominates the acquisition process. As the photon becomes scarce, eventually the signal-to-noise ratio will drop to a point that no meaningful images can be generated. Small pixel also reduces the full-well capacity which limits the dynamic range. QIS overcomes the problem by spatial-temporally oversampling the scene. In a QIS, each “image pixel” is partitioned into a group of smaller pixels called jots, and each jot is sensitive enough for photon counting. Over the past decade, numerous studies have demonstrated the feasibility of QIS, with a few prototypes developed [4, 5, 6, 7, 8]. On the image reconstruction side, various methods have also been proposed [9, 10, 11, 12].

Despite the advances in QIS hardware and image reconstruction algorithms, one important open problem is the color imaging capability of QIS. Color imaging in low light is important for many computer vision applications such as night-vision. It is also important in biological studies, where using brighter light may lead to degradation of the biological matter. While active color imaging have been demonstrated using SPAD in [13] passive color imaging using QIS is difficult for two reasons. First, the resolution of a typical single-photon detector array is not high, e.g., $320\times 240$ pixels [14], or most recently $512\times 512$ pixels   [8]. If a color filter array such as the standard Bayer pattern is implemented, the effective resolution will be halved to $160\times 120$ or $256\times 256$. For detection/tracking applications [15] this might be sufficient; but for photography the resolution could be low compared to the mainstream CMOS image sensors. Second, the color reconstruction of QIS data is different from CMOS image sensors because the QIS data is acquired under sparse-photon conditions with much lower signal-to-noise ratio. The demosaicing algorithm of QIS data needs to overcome the photon shot noise and accurately reproduce the color information. As we will discuss, QIS requires a joint reconstruction and demosaicing on top of the truncated Poisson statistics, a problem that has not been studied.

In this paper, we report the first demonstration of QIS color imaging. The QIS Pathfinder camera module developed at Gigajot Technology (as shown in Fig. 4) was used in this demonstration. The camera sensor has the same core design as the sensor described in [5], but has a Bayer pattern color filter array implemented on-chip. It has a spatial resolution of $1024\times 1024$, single-bit and multi-bit photon-counting outputs, and up to $1040$ fps speed in single-bit mode. Figure 1 shows a snapshot of the work presented in this paper. The key innovation, besides the sensor, is a new customized image reconstruction algorithm which enables joint reconstruction and demosaicing of raw QIS data. Our method leverages a few recently developed techniques in the QIS literature, including the transform-denoise method [10] and the Plug-and-Play Alternating Direction Method of Multipliers (ADMM) [16]. We demonstrate both our new sensor and the new algorithm on real data, and make quantitative evaluation based on synthetic data.

2 Megapixel quanta image sensor

When the concept of QIS was first proposed, the two mainstream design methodologies to realize single-photon detection were the electron-multiplying charge-coupled device (EMCCD) [6] and the single-photon avalanche diode (SPAD) [17, 18]. Both devices amplify the signal from a single photoelectron with a physical phenomenon called electron avalanche multiplication, where high electrical voltage (typically higher than 20V) is used to accelerate the photoelectron to produce more free electrons through collision. SPAD is more widely adopted in consumer applications because of its excellent low-noise performance and temporal resolution. The SwissSPAD [19, 20, 8] and the SPAD developed at University of Edinburgh [7, 14] are two better known examples. However due to its fabrication technology SPAD often has to make compromise in at least one of the three factors - low dark current, high quantum efficiency, and small pixel pitch. The QIS we use in this paper is based on a new type of single-photon detector proposed by Ma et al. [4, 21, 2, 5, 22]. The approach is to enhance the voltage signal generated by a single photoelectron by reducing the capacitance of the pixel output node so that the single-photon signal can overcome the background thermal noise.

Table 1 shows a comparison between a conventional CMOS image sensor, SPAD, and the prototype QIS used in this work. Among the listed different sensor parameters, the most noticeable difference between the QIS and SPADs is the small pixel pitch, where the QIS realized 1.1 $\mu$m pixel size, but SPADs typically have a pitch size around 5-20 $\mu$m. The smaller pixel size makes high spatial resolution (e.g. 10s or 100s of megapixel) possible with a small optical format (e.g. 1/4”). Another advantage of QIS over SPADs is the high quantum efficiency. Quantum efficiency measures the percentage of photons that are detectable by the sensor, which is critical for low-light imaging. SPAD cameras generally have a low quantum efficiency (e.g. $<$50%), because of low fill-factor and low photon detection capability normally called photon detection efficiency aka PDE. While low fill factor can be alleviated to an extent by using optical solutions such as microlens light-pipe. This reduces the photosensitive areal percentage in the SPAD pixels and limits the quantum efficiency. In contrast, since electron avalanche gain is not required in the QIS device, it has a quantum efficiency over 80% within the visible wavelengths.

Figure 2 shows comparison between real QIS image and a simulted CIS image. We note that we assume that the CIS no dark current. Even without considering the advantages of a lower dark current, the performance of QIS in photon starved regime is much better than the CIS.

3 QIS color imaging model

The imaging model of a color QIS is illustrated in Fig. 3. Relative to the previous QIS imaging models, e.g., [12, 10, 11], the new model considers a color filter array which allows us to select wavelengths and multiple color channels. There are a few essential components in the model.

Spatial oversampling. We represent the discretized scene image as a column vector $\boldsymbol{c}=[\boldsymbol{c}_{r};\boldsymbol{c}_{g};\boldsymbol{c}_{b}]\in[0,1]^{3N}$, where $N$ is the number of pixels, $\boldsymbol{c}_{r}$, $\boldsymbol{c}_{g}$, and $\boldsymbol{c}_{b}$ are the light intensities of the red, green and blue channels, respectively. QIS was originally designed as a spatial-temporally oversampling device which samples each pixel using $K$ jots spatially. This means every pixel has $K$ measurements, and totally there are $M\overset{\text{def}}{=}NK$ measurements. Thus, the model of this oversampling process is

[TABLE]

where $\boldsymbol{G}\in\mathbb{R}^{3M\times 3N}$ represents the $K$-fold upsampling and interpolation filtering. The constant $\alpha$ is a gain factor. In our current prototype, the oversampling factor is $K=1$ as we want to maximize the spatial resolution. In this case, we have $\boldsymbol{G}=\boldsymbol{I}$, and $M=N$.

Color filter array. To obtain color information, a color filter is placed on the top of each jot to collect light for only one of the RGB colors, and filter out the remaining two colors. This process is represented by three mutually exclusive $M\times M$ diagonal matrices or masks: $\boldsymbol{S}_{r}$, $\boldsymbol{S}_{g}$ and $\boldsymbol{S}_{B}$ for the red, green and blue channels, respectively. These matrices contain either zeros or ones on the diagonal, and $\boldsymbol{S}_{r}+\boldsymbol{S}_{g}+\boldsymbol{S}_{b}=\boldsymbol{I}$.

Incorporating the color filter array, the light exposure on the $M$ jots of QIS is represented by

[TABLE]

We call $\boldsymbol{\theta}$ the mosaiced image. If we let $\boldsymbol{S}\overset{\text{def}}{=}[\boldsymbol{S}_{r},\;\boldsymbol{S}_{g},\;\boldsymbol{S}_{b}]\in\mathbb{R}^{M\times 3M}$, then $\boldsymbol{\theta}$ and $\boldsymbol{c}$ are related by

[TABLE]

Photon arrival. The photon arrival of QIS is modeled as a Poisson process. Let $\boldsymbol{Y}\in\mathbb{N}^{M}$ be a non-negative random integer denoting the number of photons arrived at each jot during one integration period. Then, the probability of observing $\boldsymbol{Y}=\boldsymbol{y}\in\mathbb{N}^{M}$ is

[TABLE]

For single-bit QIS, we put a voltage comparator to threshold the arrived photon count into a binary decision. Thus, the read out of each jot is a binary random variable $\boldsymbol{B}\in\{0,1\}^{M}$ with $B_{m}=1$ if $Y_{m}\geq q$ and $B_{m}=0$ if $Y_{m}<q$, where $q>0$ is a threshold.

For multi-bit QIS, we use an $L$-bit quantizer to quantize the measurement. In this case,

[TABLE]

In other words, we count the photons until the number reaches a threshold. When the threshold is reached, we report the maximum number before saturation.

Assumptions. While the above model is theoretically tractable, we made three assumptions. The first is regarding the read noise associated with $\boldsymbol{Y}$. If read noise is present, the measured photon count will become $\boldsymbol{y}+\boldsymbol{\eta}$ where $\boldsymbol{\eta}$ is typically i.i.d. Gaussian. We assume that the read noise is negligible compared to the photon count, and can be eliminated by the thresholding process. We find that this is often the case with the QIS device used in this paper, which provides ultra-low read noise (0.24e- rms at room temperature) and accurate photon-counting capability. See Fig. 5 for an actual measurement of the photon count.

The second assumption we make is that the multi-bit saturation does not happen. That is, during the multi-bit sensing, we will never reach the upper threshold $q$ and so $B_{m}=Y_{m}$, which is a standard Poisson random variable. In terms of hardware, this can be achieved by implementing an automatic exposure control scheme so that the threshold $q$ is increased when the scene is too bright. We will leave this to a follow up paper as it is outside the scope of this work.

The third assumption we make is that the camera is fast enough to catch up the scene movement, and the scene remains static during the multiple temporal samplings. This allows us to utilize multiple independent measurements over time to improve the statistics. In this case, the probability distribution of $\boldsymbol{Y}$ becomes

[TABLE]

where $y_{m,t}$ is the Poisson count at pixel $m$ in time $t$.

4 QIS color image reconstruction

The task of image reconstruction is to recover color scene $\boldsymbol{c}$ from the measurements $\boldsymbol{B}$. In the gray-scale setting, we can formulate the problem as maximum-likelihood and solve it using convex optimization tools [25, 12, 9, 11]. We can also use learning-based methods, e.g., [26, 27, 28, 29] to reconstruct the signal. The method we present here is based on the transform-denoise approach by Chan et al. [10]. We prefer transform-denoise because it is a physics-based approach and is robust to different sensor configurations. For example, in learning-based approaches, if we change the number of frames to sum, we need to train a different model or neural network.

4.1 Reconstruction pipeline

The pipeline of the proposed reconstruction algorithm is shown in Fig. 6. Given the random variable $\boldsymbol{B}=\{B_{m,t}\}$ with realizations $\boldsymbol{b}=\{b_{m,t}\}$, we first sum the frames to generate a single image $\boldsymbol{Z}=\{Z_{m}\}$:

[TABLE]

In the future, this step can be integrated into the hardware of the camera, so that the output of the camera will be the sum of multiple frames. Depending on whether we are using the single-bit mode or the multi-bit mode, the statistics of $\{Z_{m}\}$ follows either a Binomial distribution or a Poisson distribution. In single-bit mode, the distribution of $\{Z_{m}\}$ is

[TABLE]

where $\Psi_{q}(\theta)\overset{\text{def}}{=}\mathbb{P}[B=0]=\sum_{j=0}^{q-1}\theta^{j}e^{-\theta}/j!$ is the probability of obtaining a zero in the binary variable $B$. As shown in [11]$,\Psi_{q}$ is the incomplete Gamma function. In multi-bit mode, since a sum of Poisson remains a Poisson, the distribution of $\{Z_{m}\}$ is

[TABLE]

If we assume that there is no processing of the bits besides temporal summation, then a simple maximum-likelihood of the probability will return us the solution:

[TABLE]

The operation $\mathcal{M}$ can be interpreted as a tone-mapping.

4.2 Variance stabilizing transform

Now, we want to process the data besides just summing the frames. However, in either single-bit or multi-bit mode, the Binomial or the Poisson statistics are not easy as the variance changes with the mean. This prohibits the use any off-the-shelf algorithms that are based on i.i.d. Gaussian assumptions, e.g., denoising or inpainting.

The variance stabilizing transform (VST) is a statistical technique that tries to alleviate the changing variance problem [30, 31, 32]. Among the different types of VSTs, we are particularly interested in the two transforms originally proposed by F. Anscombe in 1948 [33]:

[TABLE]

Here, the first transform is called the Anscombe Binomial, and the second is called the Anscombe Poisson. Both transforms are pixel-wise, meaning that we can apply the transform to each pixel independently. As proved in [10] and [33], the variance of $\mathcal{T}(\boldsymbol{z})$ is 1/4 for both the single-bit and the multi-bit case. The inverse $\mathcal{T}^{-1}$ is simply the algebraic inverse (or the unbiased inverse if we follow [34]).

The important question for color imaging is now that the photon count $\boldsymbol{z}$ is the result of applying a color filter array matrix $\boldsymbol{S}=[\boldsymbol{S}_{r},\boldsymbol{S}_{g},\boldsymbol{S}_{b}]$. That is, instead of observing the full color $\boldsymbol{\widetilde{z}}\overset{\text{def}}{=}[\boldsymbol{z}_{r};\boldsymbol{z}_{g};\boldsymbol{z}_{b}]$, we only observe a subsampled version $\boldsymbol{z}=\boldsymbol{S}\boldsymbol{\widetilde{z}}$. However, since $\mathcal{T}$ is point-wise and $\boldsymbol{S}$ is a concatenation of three non-overlapping binary matrices, it holds that

[TABLE]

Consequently, if we define $\mathcal{T}(\boldsymbol{z})$ as the observed measurement (which is available from the sensor) and treat $\boldsymbol{x}\overset{\text{def}}{=}\mathcal{T}(\boldsymbol{\widetilde{z}})$ as the unknown quantity to be determined, then the problem can be formulated as

[TABLE]

where $g$ is some regularization function controlling the smoothness of $\boldsymbol{x}$. Note that (12) is a standard demosaicing-denoising problem assuming i.i.d. Gaussian noise.

4.3 Joint reconstruction and demosaicing

The optimization problem in (12) is a standard least squares with regularization function $g$. Thus, most convex optimization algorithm can be used as long as $g$ is convex. In this paper, we adopt a variation of the alternating direction method of multiplier (ADMM) by replacing $g$ with an off-the-shelf image denoiser. Such algorithm is coined the name Plug-and-Play (PnP) [35] (and different versions thereafter [16]).

For the particular problem in (12), the PnP ADMM algorithm iteratively updates the following two steps:

Demosaicing Module:

[TABLE]

Denoising Module:

[TABLE]

and updates the Lagrange multiplier by $\boldsymbol{u}^{(k+1)}=\boldsymbol{u}^{(k)}-(\boldsymbol{x}^{(k+1)}-\boldsymbol{v}^{(k+1)})$. Readers interested in the detailed derivation can consult, e.g., [16]. Here, $\rho$ is an internal parameter that controls the convergence. The operator $\mathcal{D}$ is an off-the-shelf image denoiser, e.g., Block-matching and 3D filtering (BM3D) or deep neural network denoisers. The subscript $\rho/\lambda$ denotes the denoising strength, i.e., the hypothesized “noise variance”. Since $\boldsymbol{S}^{T}\boldsymbol{S}$ is a diagonal matrix, the inversion is pointwise.

4.4 Non-iterative algorithm

The $1.1\mu$m pixel pitch of the proposed QIS can potentially lead to a spatial resolution as high as or even higher than a conventional CMOS sensor. When this happens, in certain applications we can trade-off the color reconstruction efficiency and the resolution. For example, instead of using one jot for one pixel, we can use four jots for one pixel as shown in Fig. 7.

Using four jots for one pixel allows us to bypass the iterative ADMM steps because there is no more missing pixel problem. In this case, the matrix $\boldsymbol{S}\in\mathbb{R}^{M\times 3M}$ will become $\boldsymbol{S}=\mbox{diag}\{\frac{1}{4}\boldsymbol{I},\frac{1}{2}\boldsymbol{I},\frac{1}{4}\boldsymbol{I}\}\in\mathbb{R}^{3M\times 3M}$, and hence (12) is simplified to a denoising problem with different noise levels for the three channels. In particular, the green channel has half of the variance of the red and the blue. For implementation, we can modify a denoiser, e.g., BM3D to accommodate the different noise variances. Since there is no more ADMM iteration, the algorithm is significantly faster. While we have used BM3D to demonstrate the results, any off the shelf denoiser which is used in CIS based cameras can be used to denoise the image for the four-jot to one-pixel method. We would also like to stress on the fact that both the Anscombe transform and the transform $\mathcal{M}$ can be implemented as a look up table. So, this method can be as fast a current denoiser being used in a CIS based camera.

4.5 Comparisons

We compare the proposed method with several existing methods on a synthetic dataset shown in Fig. 8. We simulate the raw color QIS data by assuming a Bayer pattern and using the image formation pipeline described in the previous section. We demosaic the images using: (a) a baseline method using MATLAB’s demosaic preceded by gray-scale BM3D denoising of $R$, $G_{1}$, $G_{2}$ and $B$ channels and followed by color BM3D denoising; (b) Least-squares luma-chroma demultiplexing (LSLCD) method [36], which has a built-in BM3D denoiser; (c) Hirakawa’s PSDD method [37, 38], which does joint denoising and demosaicing for Poisson noise; and (d) the proposed method using BM3D with $(\lambda,\rho)=(0.001,5)$. We apply variance stabilizing transform, except for PSDD which is designed for Poisson noise. The results show that the proposed method has a significantly better performance, both in terms of Peak Signal to Noise Ratio (PSNR) and visual quality.

5 Experimental results

5.1 Synthesized QIS data

We conduct a synthetic experiment to provide a quantitative evaluation of the performance of the proposed algorithm. To this end, we simulate the image formation pipeline by passing through color images to generate the QIS raw input data, with different number of bits. Figure 9 shows one example, and in the Supplementary Report we have additional examples.

In our simulation, we assume that the number of QIS frames is $T=4$, and the average number of photon per pixel is $0.28$, $0.85$, $1.98$ and $4.23$ photons / frame for 1-bit, 2-bit, 3-bit and 4-bit QIS, respectively. On the measurement side, we generate single-bit and multi-bit data by thresholding the raw sensor output. To reconstruct the image, we use the proposed method with PnP and BM3D. The parameters are set as $\rho=1$ and $\lambda=0.007,0.003,0.002$ and $0.0007$ for 1-bit, 2-bit, 3-bit and 4-bit QIS, respectively. With as low as 1-bit, the reconstructed image in Fig. 9 is already capturing most of the features. As the number of bits increases, the visual quality improves.

5.2 Real QIS data

We first show the reconstruction of an image of the Digital SG $\mathrm{ColorChecker}^{\footnotesize\textregistered}$ chart. We generate two sets of measurements: (a) a set of 50 one-bit frames, quantized with a threshold $q=4$, and (b) a set of 10 five-bit frames. We use PnP and BM3D with $(\lambda,\rho)=(0.15,10)$ and $(0.01,10)$ for the 1-bit and 5-bit data, respectively. After reconstruction, the results are multiplied by a $3\times 3$ color correction matrix to mitigate any color cross-talk. This matrix is generated by linear least square regression of the 24 Macbeth color patches that lies inside the SG ColorChecker chart. The results in Fig. 1 suggest that while 1-bit mode has color discrepancy with the ground truth, the 5-bit mode is producing a reasonably-high color accuracy.

Next, we show the result of imaging real scenes. See Fig. 10. In this experiment, the exposure time for each frame is set to 50$\mu$s. The average number of photons per pixel is approximately 4.2 photons for the ”QIS” sign image, 3 photons for the Pathfinder image, 1.9 photons for the duck image, and 2.9 photons for the mushroom image. For all images, we collect the data using a 5-bit QIS.

We demonstrate two algorithms: (i) the proposed transform-denoise framework using PnP + BM3D, and (ii) assuming four-jots to one-pixel scenario by trading half of the spatial resolution. The typical runtime on an unoptimized MATLAB code is approximately 4 minutes for PnP, and 10 seconds for the four-jot to one-pixel method. An interesting observation is that even in the lower-resolution case, the details are not significantly deteriorated unless we zoom-in. However, the speed up we get is substantial.

6 Conclusion and discussion

We demonstrated the first color imaging for a mega-pixel Quanta Image Sensor (QIS). We designed a joint reconstruction-demosaicing algorithm for the quantized Poisson (and Poisson) statistics of the sensor data. Our solution involves a variance stabilizing transform and an iterative (and a non-iterative) algorithm. By integrating the sensor and the new algorithm, the sensor is able to acquire color images at a photon level as low as a few photons per pixel, a level that would be difficult for standard CMOS image sensors.

7 Acknowledgement

The authors thank Prof. Eric Fossum for many valuable feedback. The 1Mjot color Quanta Image Sensor used in the demonstration was designed by J. Ma, S. Masoodian, and D. Starkey at Dartmouth College Advanced Image Sensor Laboratory led by Dr. Eric Fossum. The sensor was manufactured by Taiwan Semiconductor Manufacturing Company (TSMC). The camera module developed at Gigajot Technology was partially supported by National Science Foundation SBIR program. A Gnanasambandam, O. Elgendy and S. H. Chan are supported, in part, by the U.S. National Science Foundation under Grant CCF-1718007.