Experimental Verification of PCH-EM Algorithm for Characterizing DSERN Image Sensors
Aaron Hendrickson, David P. Haefner, Nicholas R. Shade, Eric R. Fossum

TL;DR
This paper experimentally verifies the PCH-EM algorithm's effectiveness in characterizing DSERN image sensors, demonstrating its accuracy across various exposure and noise levels and validating the PCD model's predictive capability.
Contribution
It provides a comprehensive experimental validation of the PCH-EM algorithm for DSERN sensors and confirms the PCD model's accuracy in predicting sensor behavior.
Findings
PCH-EM accurately characterizes DSERN pixels across a wide range of conditions.
The PCD model effectively predicts the ensemble distribution of the sensor.
Experimental results align well with model predictions, confirming their applicability.
Abstract
The Photon Counting Histogram Expectation Maximization (PCH-EM) algorithm has recently been reported as a candidate method for the characterization of Deep Sub-Electron Read Noise (DSERN) image sensors. This work describes a comprehensive demonstration of the PCH-EM algorithm applied to a DSERN capable quanta image sensor. The results show that PCH-EM is able to characterize DSERN pixels for a large span of quanta exposure and read noise values. The per-pixel characterization results of the sensor are combined with the proposed Photon Counting Distribution (PCD) model to demonstrate the ability of PCH-EM to predict the ensemble distribution of the device. The agreement between experimental observations and model predictions demonstrates both the applicability of the PCD model in the DSERN regime as well as the ability of the PCH-EM algorithm to accurately estimate the underlying model…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCCD and CMOS Imaging Sensors · Electron and X-Ray Spectroscopy Techniques · Infrared Target Detection Methodologies
Experimental Verification of PCH-EM Algorithm for Characterizing DSERN Image Sensors
Aaron Hendrickson, David P. Haefner, Nicholas R. Shade , and Eric R. Fossum
Abstract
The Photon Counting Histogram Expectation Maximization (PCH-EM) algorithm has recently been reported as a candidate method for the characterization of Deep Sub-Electron Read Noise (DSERN) image sensors. This work describes a comprehensive demonstration of the PCH-EM algorithm applied to a DSERN capable quanta image sensor. The results show that PCH-EM is able to characterize DSERN pixels for a large span of quanta exposure and read noise values. The per-pixel characterization results of the sensor are combined with the proposed Photon Counting Distribution (PCD) model to demonstrate the ability of PCH-EM to predict the ensemble distribution of the device. The agreement between experimental observations and model predictions demonstrates both the applicability of the PCD model in the DSERN regime as well as the ability of the PCH-EM algorithm to accurately estimate the underlying model parameters.
Index Terms:
conversion gain, DSERN, EM algorithm, PCH, PCH-EM, photon counting, QIS, quanta exposure, read noise.
I Introduction
As the detection precision of advanced camera technology improves, the ability to properly characterize and evaluate modern image sensors only becomes more important. While the traditional Photon Transfer (PT) method [1, 2, 3] can be applied to Deep Sub-Electron Read Noise (DSERN) image sensors, it has been shown there are other algorithms that can improve the accuracy and precision of the camera characterization [4, 5, 6]. Specifically, both the Photon Counting Histogram (PCH) method [4, 7, 8, 9] and recently introduced Maximum Likelihood Estimation (MLE) based method [5] have been demonstrated to incur less uncertainty in their estimates as compared to the PT method. Recently, Hendrickson and Haefner proposed a fourth method, Photon Counting Histogram Expectation Maximization (PCH-EM), that improves on these techniques by providing an automated algorithm for simultaneous maximum likelihood estimation of quanta exposure, conversion gain, bias (DC offset), and read noise of DSERN pixels from a single sample of data [6].
Due to the cutting edge nature of DSERN capable sensors, the PCH-EM algorithm was initially demonstrated using simulated Monte Carlo experiments. In this paper, through the use of an early photon-counting-capable Quanta Image Sensor (QIS) from Gigajot Technology Inc., a more comprehensive demonstration of the PCH-EM algorithm and verification of the associated Photon Counting Distribution (PCD) model is provided. This is accomplished by first reviewing the assumed mathematical model and theoretical framework behind the PCH-EM method. New theory pertaining to ensemble statistics of DSERN sensors is also introduced. Experimental conditions and data capture methods needed for dark current characterization with PCH-EM are provided. The experimental observations are evaluated through the PCH-EM algorithm, providing a full characterization of the sensor giving per-pixel estimates of dark current, conversion gain, bias, and read noise all from a single sequence of images captured under dark conditions. The per-pixel characterization results are then combined with the PCD model to predict the ensemble distribution for the device, showing that the model is able to predict the distribution of the raw sensor data. This agreement demonstrates both the applicability of the PCD model in the DSERN regime as well as the PCH-EM algorithm’s ability to accurately estimate the underlying model parameters.
II Theory
II-A The PCD
The digital output of a DSERN pixel is modeled as
[TABLE]
where is the quanta exposure , the input referred analog read noise , the conversion gain , is the pixel bias , and denotes rounding to the nearest integer. As such, the random variable represents the random process of adding noise to a number of electrons followed by the application of gain, offset, and finally quantization. Note that this is a general sensor model not specific to DSERN devices. What differentiates DSERN pixels is the fact that the signal corrupting noise is sufficiently small so that the electron number can be reasonably estimated.
Assuming , quantization (rounding) in (1) can be modeled as an additive noise component so that the distribution of is reasonably approximated by the Photon Counting Distribution (PCD) [6]
[TABLE]
where are the PCD parameters and is the Gaussian probability density with mean and variance . In (2), is the combined read and quantization noise in .
II-B The PCH-EM Algorithm
Given a random sample with and an initial estimate of the parameters , the PCH-EM algorithm iteratively updates the parameter estimates via the update equations [6]
[TABLE]
where and are the first two sample moments and
[TABLE]
where
[TABLE]
are the so-called membership probabilities; representing a probability distribution of the unknown electron number associated with each observation . As such, the membership probabilities satisfy .
In each iteration, the algorithm takes the current estimate and then performs an Expectation (E) step to compute the followed by a Maximization (M) step to update the estimate according to (3). In doing so, the algorithm guarantees an increase in the likelihood of the sample at each iteration such that a local maxima of the likelihood function is always achieved111Assuming the starting point is sufficiently good, PCH-EM achieves the global maximum of the likelihood function so that the final estimates are maximum likelihood estimates for their respective parameters. [10]. The algorithm halts when a specified convergence criteria is met.
In the context of machine learning, the general EM algorithm can be viewed as a density-based clustering algorithm, assigning labels to each datapoint based on what cluster the datapoint is most likely to belong to. In the context of PCH-EM, the Gaussian components comprising the PCD are the clusters, with the electron number determining which cluster an observation belongs. As such, a natural byproduct of the PCH-EM algorithm is the ability to map each observation to a nonnegative integer representing a best estimate for the electron number associated with each observation. This post-process denoising of the sensor data is accomplished by applying the membership probabilities via
[TABLE]
In essence, the mapping described in (6) is clustering the data by its mostly likely electron number in an optimal manner as to reduce bit error rates [11].
To see this optimal clustering in action, consider the example , where the values of the parameters and are selected so that the data can be interpreted as being in units of . Figure 1 shows the PCD along with the optimal cluster edges obtained from the quantization described in (6). While the PCD peaks are centered at nonnegative integers, it can be seen that the cluster edges are not directly centered between the peaks nor are the clusters of equal size. This nonuniform clustering ensures optimal estimation of the electron number for each observation.
II-C Ensemble Statistics
The PCD in (2) describes the distribution of data produced by a single DSERN pixel. When considering data produced by an array of DSERN pixels, each with potentially different parameters, the parameters themselves can be modeled as random variables. Denoting as the random variable describing the ensemble of pixels leads to the hierarchical model
[TABLE]
so that the distribution of is given by the Ensemble PCD (EPCD)
[TABLE]
where is the joint distribution of the parameters (with corresponding joint density ) and is the parameter space denoting all possible values of the parameter vector . Alternatively, the EPCD can be written as , where denotes the expected value w.r.t. . The moments of the EPCD can be given in terms of the moments of the parameters as shown in Appendix A.
Unlike the per-pixel PCD, the peaks (local maxima) in the EPCD typically disappear at higher signal levels (c.f. Figure 4 (top)), which is indicative of conversion gain nonuniformity (see Appendix B). For this reason, it is also useful to consider the ensemble distribution after correcting conversion gain nonuniformity and bias through a conventional two point Non-Uniformity Correction (NUC). Applying a gain and offset correction both improves the resolution of the peaks and centers them on the nonnegative integers. The Non-Uniformity Corrected (NUCed) EPCD can be found by setting and as constants leading to the model
[TABLE]
with distribution
[TABLE]
where is the total read and quantization noise in units of electrons and . Examples of both the EPCD and NUCed EPCD can be seen in Figure 4 (see Section IV-C).
Lastly, consider the ensemble distribution of the electron number , which will appear later when evaluating the ability of PCH-EM to predict electron numbers. On a per-pixel basis the electron number is Poisson distributed and since each pixel may have a unique quanta exposure, the ensemble electron number is described by
[TABLE]
with probability mass
[TABLE]
where denotes the moment generating function of the quanta exposure random variable .
III Experimental Method
The experimental data was collected using a developmental DSERN capable camera from Gigajot Technology Inc. The specific camera chosen is the GJ00111, which consists of a monochrome one megapixel CMOS QIS with pitch pixels. It was operated at its full bit-depth of 14-bits using four Correlated Multi-Sample (CMS) cycles to minimize read noise.
For this experiment, the PCH-EM algorithm was used to estimate per-pixel dark current. This is accomplished through operating the camera with a lens cap and using a long integration time of . The long integration time ensures each pixel was given ample opportunity to produce thermally generated free-electrons. A total of frames over a region of interest were captured continuously under the dark environment.
IV Results
The PCH-EM algorithm code, available on the Mathworks File Exchange [12], was applied on a per pixel basis to the experimental dataset. To expedite calculations, parallel methods (memory limitations permitting) can be used as the per-pixel estimates can be found independently. An additional speed improvement is also possible by implementing PCH-EM through histograms (number of occurrences for each unique DN observed). Using the histogram is especially beneficial when there are relatively few unique DN values in a sample compared to the number of frames reported. Finally, an additional improvement can also be achieved by vectorizing the code, running the same sequence of calculations on multiple pixels simultaneously with MATLAB’s optimized methods. The final time for the analysis was slightly under an hour running 12 cores on the machine used. Eventually, the release of the optimized histogram implementation of PCH-EM and other improvements to the algorithm in future updates will be provided on the Mathworks file exchange.
The experiments were conducted with no external illumination with the intention of characterizing the dark current. The dark current given in units of is found from the relation . Additionally, the read plus quantization noise given in units of is found by multiplying the gain by the square root of , i.e. .
IV-A Per-pixel Characterization
The PCH-EM algorithm provides estimates of the PCD model parameters , , , and . Using these estimated parameters, the predicted probability density of the individual pixels can be computed. A comparison of the predicted density (solid black line) against the observed experimental histogram (gray bars) for four of the sensor’s pixels are shown in Figure 2.
The four pixels shown were selected to demonstrate that the algorithm provides a good fit to the data at high or low quanta exposure as well as high or low read noise. Note that with low quanta exposure, there are very few peaks that may be used for estimating the conversion gain. However, even under such conditions, the probability density function calculated from the parameter estimates still accurately matches the observed data histogram. Also note that since these are dark frame measurements, the quanta exposure represents the expected number of free-electrons generated per-integration time via thermal contributions. Since this is proportional to the integration time, increasing the integration time will increase the observed quanta exposure and can further improve the estimates of the conversion gain if needed.
IV-B PCD Parameter Maps
Applied to the array, the PCD parameters for each pixel were estimated resulting in four two-dimensional arrays (maps) containing per-pixel estimates of , , , and . Perhaps the most important for DSERN sensors is the distribution of read noise shown below in Figure 3. As can be seen, the vast majority of the pixels have an estimated read noise of less than with the median of the histogram occurring at .
The spatial context and distributions of other parameters are found in Figures 6-9 (see Appendix C). Structure (or lack thereof) observed in the parameter maps can be tied back to the architecture of the sensor and may potentially be useful in tuning the sensor parameters during development.
IV-C Ensemble Distributions
Applying the parameter estimates through (8), one can observe how the PCH-EM algorithm fits the sensor data on the array scale by predicting the EPCD of the sensor and comparing it to the ensemble histogram of the raw data. In order to estimate the EPCD, the unknown joint density of the PCD parameters must be determined. While this density is unknown, it may be approximated by binning the four parameter maps in a four-dimensional histogram. After normalization, this provides a discrete approximation for . The approximate EPCD is then found by evaluating (8), replacing integrals with sums, for an appropriate range of -values.
Figure 4 (top) shows the ensemble histogram made from frames of the raw experimental data compared to the estimated EPCD using the parameter maps. For comparison, two EPCD’s were estimated under the assumption of mutually independent parameters ( approximated by the product of four individual histograms) and dependent parameters ( approximated by a single four-dimensional histogram), respectively. One can see that the EPCD under the assumption of dependent parameters provides an excellent fit to the raw data; thus providing experimental confirmation of the PCD model and PCH-EM algorithm. The fact that the estimated EPCD for dependent parameters () provides a better fit compared to the case of independent parameters () makes sense since, for example, the expression for the variance contains ; therefore, it is expected for and to be dependent. In the ensemble histogram, it can be observed that the peaks become less distinct as signal increases which is usually indicative of conversion gain nonuniformity (see Appendix B).
Figure 4 (bottom) shows the NUCed ensemble histogram found by subtracting per-pixel estimates of from each frame and then multiplying the bias corrected frames by the per-pixel estimates of . This effectively removes the effects of gain and offset nonuniformity from the raw data. Notice that the peaks are now more clearly resolved and located at nonnegative integers showing that this two-point NUC restores the electron counting capabilities of the sensor. Using the same approach as before, the NUCed EPCD can be found by approximating the joint density from the quanta exposure and read noise maps under the assumption of dependent and independent parameters, and then approximating the double integral in (10) by sums. As seen in the bottom of Figure 4, both ensemble predictions under the assumption of dependent parameters () and independent parameters () fit the NUCed data quite well with a slight advantage given to the case of dependent parameters. This indicates that the quanta exposure (dark current) is nearly independent of the read noise (when in units of electrons), which is to be expected (see discussion at the end of Appendix B).
Through Figures 2 and 4, the PCD and PCH-EM algorithm have been shown to be effective in modeling DSERN sensor data and providing estimates of the model parameters, respectively. What remains to be demonstrated is if the electron number prediction formula in (6) can effectively recover the electron numbers for each observation. Using (6), the predicted electron number for each pixel of the -frame stack of raw experimental data was computed. This process resulted in an array of nonnegative integers, the same size as the image stack, containing the all predictions. A histogram of the predictions is given in Figure 5. While it cannot be known if these predictions agree with the actual electron numbers associated with each observation, the distribution of the predictions can be compared to what would be expected according to the ensemble electron number probability mass in (12). To predict this ensemble distribution, the unknown quanta exposure density was approximated by binning the quanta exposure map and then replacing the integral in (12) by a finite sum. Figure 5 compares the ensemble histogram of the electron number predictions against the predicted probability mass according to the model. Recalling the discussion in Section II-B, the data presented in Figure 5 can be viewed as an optimal quantization of the NUCed EPCD in Figure 4 (bottom). The quality of fit between the data and predicted probability mass demonstrates, at the very least, that the predicted electron numbers agree with the actual electron numbers in terms of distribution.
V Discussion and Future Work
In this paper, the PCH-EM algorithm proposed in [6] was successfully demonstrated to accurately estimate quanta exposure, conversion gain, bias, and read noise of DSERN pixels in an automated fashion. Combining the assumed model with the corresponding estimated parameters accurately recreates the raw sensor data histograms, both on a per-pixel level as well as at the ensemble (array) level. The ensemble prediction required accounting for the correlation of the four model parameters. Additionally, it was shown how a two-point non-uniformity correction may be determined and applied to the ensemble, which improves the resolution of individual electron peaks and restores electron counting of the device. Lastly, the ability of PCH-EM to denoise raw sensor measurements and recover the hidden electron signal was demonstrated.
This PCH-EM algorithm is a powerful tool for investigating and tuning the performance of DSERN sensors, as it can be applied automatically over a large span of parameters. Through the use of the estimated parameter maps, PCH-EM not only is useful for sensor characterization but also may find application during the advanced development of the sensors themselves. Also, together with the Monte Carlo methods provided in [12], an experimentalist can investigate the number of frames required to achieve a desired uncertainty.
Future work will include expanding upon the current method to combine multiple illumination level measurements in a multi-sample version of PCH-EM, exploring techniques for accounting for non-linear responses, and releasing optimized code on the Mathworks File Exchange. Additionally, implementing various techniques for estimating the sample Fisher information will be pursued [13, 14, 15, 16]. The ability to estimate the Fisher information would allow the PCH-EM algorithm to not only provide the parameter estimates but also their uncertainties. Ultimately, a generalized characterization method should work across the full dynamic range of the sensor and full parameter space of photon counting sensors.
VI Acknowledgments
The authors would like thank Nico Schlömer for his matlab2tikz function, which was used to create the figures throughout this work [17].
Appendix A Moments of the Ensemble Distributions
Moments of the EPCD are found by noting that . Using the law of total expectation the first moment is
[TABLE]
Likewise, by the law of total variance
[TABLE]
Expanding the first variance term further then gives the final result of
[TABLE]
The analogous moments of the NUCed EPCD come from these expressions upon setting and as constants. This gives and .
Appendix B Dependence of Ensemble Peak Resolution on Parameter Nonuniformity
Here, the loss of peak resolution in the EPCD at higher signal levels and the dependence of this phenomenon on parameter nonuniformity is investigated. To do this, it is important to first understand why this behavior is not observed in the single pixel model.
Recall the distribution for a single pixel is given by the PCD
[TABLE]
which is comprised of an infinite mixture of Gaussian components given by the probability density . The individual components are thus isolated by considering the distribution of the random variable . Computing the variance of this conditioned variable gives
[TABLE]
which is independent of the electron number . This means that the widths of each component making up the PCD are the same; thus, as signal increases, the resolution of individual peaks remains constant.
Repeating this calculation for the ensemble variable , while assuming the appropriate regularity conditions to allow interchanging series and integration, the EPCD in (8) can be written in the form
[TABLE]
which is comprised of an infinite mixture of non-Gaussian components given by the probability density . The variance of the conditioned variable is then given by
[TABLE]
where
[TABLE]
and
[TABLE]
Upon inspection, is dependent on ; thus the widths of the components comprising the EPCD vary with signal level leading to a loss of peak resolution at higher signals.
What is not clear is if the dependence of on is linked to the nonuniformity of only a subset of the parameters. This can be investigated by considering what happens to when holding none, one, two, three, or all four parameters constant. This results in sixteen cases. Evaluating all sixteen cases, it can be shown that holding , , , , or constant removes the dependence on . Since the case implies the and cases, and the case results in the original PCD, there are only two ways for to be independent of under dependent parameters: when is constant or is constant. Thus holding certain subsets of the parameters constant does remove the dependence of on resulting in constant peak resolution. It is also worth noting that appears in all of these cases showing that if conversion gain nonuniformity exists, then the EPCD component width must depend on ; causing peak resolution to decrease at higher signal levels. It is interesting that holding only constant does not remove the dependence on ; however, note that if is independent of and then is held constant . This shows that the loss of resolution in the EPCD peaks can be solely contributed to conversion gain nonuniformity of is independent of . With so many combinations to consider, a study of the statistical dependence of the individual parameters in actual sensor systems may help guide further analysis.
The component width of the NUCed EPCD can also be found as a special case of for and constant. This special condition leads to
[TABLE]
where the last equality holds when is independent of . This explains why the component width of the NUCed EPCD in Figure 4 appears to be constant.
Appendix C Estimated Parameter Maps
One of the challenges when displaying the estimated parameter maps is the presence of outliers, which given a limited dynamic range of the display means one typically has to clip the map values. To provide a visually aesthetic way to display the maps, a nonlinear transformation of the form
[TABLE]
was applied to the map elements. Here, is the quantile function, is the empirical cumulative distribution function of the map, and is the th element of the map. This transformation takes the original histogram of the map and shapes it into that of a distribution; however, because this transformation is monotone, any structures in the original map are carried over to the final transformation, all while suppressing the appearance of outliers so that the structure is clearly observed.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] B. P. Beecken and E. R. Fossum, “Determination of the conversion gain and the accuracy of its measurement for detector elements and arrays,” Appl. Opt. , vol. 35, no. 19, pp. 3471–3477, Jul 1996.
- 2[2] J. R. Janesick, Photon Transfer: D N → λ → 𝐷 𝑁 𝜆 DN\to\lambda . SPIE, 2007.
- 3[3] A. Hendrickson, D. P. Haefner, and B. L. Preece, “On the optimal measurement of conversion gain in the presence of dark noise,” J. Opt. Soc. Am. A , vol. 39, no. 12, pp. 2169–2185, Dec 2022.
- 4[4] D. A. Starkey and E. R. Fossum, “Determining conversion gain and read noise using a photon-counting histogram method for deep sub-electron read noise image sensors,” IEEE Journal of the Electron Devices Society , vol. 4, no. 3, pp. 129–135, May 2016.
- 5[5] K. Nakamoto and H. Hotaka, “Efficient and accurate conversion-gain estimation of a photon-counting image sensor based on the maximum likelihood estimation,” Opt. Express , vol. 30, no. 21, pp. 37 493–37 506, Oct 2022.
- 6[6] A. Hendrickson and D. P. Haefner, “Photon counting histogram expectation maximization algorithm for characterization of deep sub-electron read noise sensors,” Cornell University ar Xiv , vol. 2302.00090, 2023.
- 7[7] J. Ma, D. Starkey, A. Rao, K. Odame, and E. R. Fossum, “Characterization of quanta image sensor pump-gate jots with deep sub-electron read noise,” IEEE Journal of the Electron Devices Society , vol. 3, no. 6, pp. 472–480, 2015.
- 8[8] J. Ma and E. R. Fossum, “Quanta image sensor jot with sub 0.3e- r.m.s. read noise and photon counting capability,” IEEE Electron Device Letters , vol. 36, no. 9, pp. 926–928, 2015.
