Fourier Ptychographic Neural Network Combined with Zernike Aberration Recovery and Wirtinger Flow Optimization

TL;DR

This paper introduces a new imaging method that improves high-resolution image quality by correcting optical aberrations using a neural network and advanced optimization techniques.

Contribution

The novel contribution is a neural network model that integrates Zernike aberration recovery and optimized Wirtinger flow for improved Fourier ptychographic imaging.

Findings

The proposed model effectively improves aberration correction accuracy in Fourier ptychographic microscopy.

The method maintains good correction performance even in complex imaging scenarios.

Optical aberration's impact on image quality is significantly reduced using the new approach.

Abstract

Fourier ptychographic microscopy, as a computational imaging method, can reconstruct high-resolution images but suffers optical aberration, which affects its imaging quality. For this reason, this paper proposes a network model for simulating the forward imaging process in the Tensorflow framework using samples and coherent transfer functions as the input. The proposed model improves the introduced Wirtinger flow algorithm, retains the central idea, simplifies the calculation process, and optimizes the update through back propagation. In addition, Zernike polynomials are used to accurately estimate aberration. The simulation and experimental results show that this method can effectively improve the accuracy of aberration correction, maintain good correction performance under complex scenes, and reduce the influence of optical aberration on imaging quality.

Figures

Full text

1. Introduction

Fourier ptychographic microscopy (FPM) [1,2] is an emerging imaging technique, which was proposed by Zheng et al. in 2013. Compared to the traditional microscopy imaging mode, this technique combines the ideas of phase recovery [3,4,5], stacked imaging [6], and synthetic aperture [7] by breaking through the limitation of the numerical aperture of the objective lens and improving the image resolution under the premise of ensuring the original size of the field of view. However, optical aberration emerges in the actual application process, which imposes certain limitations on the imaging results.

Aberration refers to the difference between the actual and ideal images. While beam focusing in optics can be elaborated as the convergence of light rays to a single point, aberration is the deviation of light rays from the optimal focal point, causing the focus to spread in space [8]. As the imaging system has a certain aperture and field of view, the imaging position for incident light can be different at different apertures. In optics, the aberration in the imaging system can be divided into seven kinds, namely, spherical aberration, coma, dispersion, field curvature, aberration, positional chromatic aberration, and magnification chromatic aberration, as shown in Figure 1.

Aberration is commonly corrected by restoring high-resolution complex objects and unknown aberration pupil functions in the iterative process. For example, Ou et al. [9] proposed a phase recovery algorithm (EPRY-FPM) based on the ePIE method [10], which restores the extended sample spectrum and the pupil function of the imaging system by employing the image of the sample captured by the FPM. With the continuous development of deep learning, more and more researchers have introduced the aberration correction process into the neural network, with the purpose of improving the computational efficiency of the algorithm by taking advantage of its fast computing. For example, using the neural network, Zhang et al. [11] modeled the samples and aberrations as the learnable weights of the multiplication layer and discovered that the INNM network architecture could obtain a complex sample without aberrations. Zhang et al. [12] proposed a Fourier imaging neural network (FINN-CP) with Tensorflow, which is composed of two models, for effectively correcting the position error and wavefront aberration of the system. Hu et al. [13] proposed a microscopic image aberration correction method based on deep learning and aberration prior knowledge, which enhances and corrects the microscopic image in the form of image restoration. Zhang et al. [14] combined the channel attention module with a physics-based neural network to adaptively correct aberrations; Zhao et al. [15] established the relationship between the phase and aberration coefficient through deep learning to segment samples and backgrounds [16] and realized fast automatic aberration compensation correction [17]. Wu et al. [18] proposed an FPM aberration correction reconstruction framework (AA-P) algorithm based on an improved phase retrieval strategy, which improves the iterative reconstruction quality by optimizing the spectral function and the pupil function update strategy while alleviating the influence of mixed wavefront aberrations on the reconstructed image quality and avoiding the occurrence of errors in the reconstruction process. The quality of image reconstruction can be ensured by aberration correction, endowing the reconstructed image with more details. Xiang et al. [19] proposed a phase diversity-based FP (PDFP) scheme for aberration correction. The PD algorithm is an unconventional imaging technique introduced by Gonsalves and Chidlaw [20], which characterize wavefront aberrations by means of a set of focused images and defocused images. Experiments have proven the ability of this scheme to correct changing aberrations and improve image quality. Aberration correction can ensure the quality of image reconstruction, achieving the reconstructed image with more details.

In this paper, we propose an aberration correction method based on the Fourier ptychographic microscopy technique for the aberration existing in the imaging process and name it Integrated Neural Network based on Improved Wirtinger Flow (INN_IWF). The model proposed in this paper is a trainable network constructed on the basis of the TensorFlow framework to simulate the entire process. The network simulates the forward imaging process of the Fourier ptychographic microscopy system while modelling the optical aberration of the objective lens as the optical pupil function to better estimate the optical aberration and optimize the update by back propagation. Furthermore, the alternate updating (AU) mechanism and the Zernike mode are introduced to the model to further improve the performance of the proposed network. Therefore, this method can effectively recover optical aberrations while guaranteeing the overall performance of the network. The results of several sets of experiments show that the mentioned method is superior to other methods in its capability to effectively improve the quality of image reconstruction while retaining more detailed information.

2. Methods

2.1. Fourier Ptychographic Microscope

The difference between a Fourier ptychographic microscope and conventional microscope is that the Fourier ptychographic microscope uses an array of LEDs instead of a conventional microscope light source. The LEDs are correctly selected to achieve illumination from a variety of angles. Figure 2a shows the RX50 series upright field microscopes and Figure 2b shows the simulation schematic diagram of the device. The camera used in the device is a DMK 33UX264 camera (The Imaging Source, Bremen, Germany, 3.45 µm, 2448 × 2048). The purpose of this device is to digitally image the sample. Optical imaging collected by the device can be accomplished either directly visually or by using the software to view the actual iPlease provide manufacturer and address informationmages captured by the camera.

The device consists of a DMK 33UX264 camera, an eyepiece, an optical path selector lever, a Y-axis moving handwheel, a mirror group, a tightening and loosening adjusting handwheel, an adjusting light wheel, a light collector mirror, an X-axis moving handwheel, a mechanical platform, an LED light board holder, an LED light board (20 × 20), etc. The LED light board parses the commands from the MATLAB program sent through the serial port to light up the LED lights in the specified positions. With LED lights and LED built-in RGB three-color beads, the device can capture images using a black and white camera and synthesize these images into color images. The light panel can be fixed or moved downwards and upwards through LED brackets.

2.2. Imaging Model and Reconstruction Model

2.2.1. Imaging Model

In the forward imaging process, the sample can be represented by the transfer function $[eqn]$ , where r represents the two-dimensional coordinate. Assuming that the distance between the LED lamp and the sample is far enough, the illumination wave of the LED lamp can be approximated as an oblique plane wave, and the wave vector of the nth lamp can be expressed as

$$[eqn]$$

where $[eqn]$ represents the incident angle of the nth LED lamp, $[eqn]$ is the wavelength of the incident light, and the complex amplitude entering the sample plane is expressed as $[eqn]$ . When the nth LED lamp illuminates the sample, the output field after Fourier transform can be expressed as $[eqn]$ . Illuminating the sample using the oblique plane wave with a wave vector is equivalent to the shift $[eqn]$ of the sample spectrum $[eqn]$ . When passing through the objective lens, the field is lowpass filtered by the pupil function $[eqn]$ . At this time, the forward imaging process of FPM can be expressed as

$$[eqn]$$

where $[eqn]$ represents the intensity information on the sensor, $[eqn]$ represents the complex amplitude distribution on the sensor, $[eqn]$ represents the sample spectrum illuminated by its plane wave vector $[eqn]$ plane wave, $[eqn]$ represents the two-dimensional coordinate, and $[eqn]$ represents the inverse Fourier transform [22].

2.2.2. Reconstruction Model

In the reconstruction process, FPM obtains a high-resolution complex amplitude distribution $[eqn]$ by synthesizing images with different frequency domain information. The classical FPM reconstruction algorithm iteratively estimates the complex amplitude image and updates it using the captured intensity image. An iteration can be expressed as

$$[eqn]$$ $$[eqn]$$

Equation (3) is used to estimate the high-resolution image relative to each LED light, while Equation (4) updates the high-resolution image by utilizing the captured low-resolution intensity image. The degree of spectral convergence can be known by repeated calculations, and low-resolution images can be used for the initial $[eqn]$ . Finally, the estimated spectral $[eqn]$ is transformed into $[eqn]$ by inverse Fourier transform, and the high-resolution image is extracted from $[eqn]$ .

2.3. Integrated Neural Network Based on Improved Wirtinger Flow

2.3.1. Network Architecture

The whole network implements aberration correction in the Tensorflow framework. Figure 3 shows the overall flowchart of INN_IWF. The sample images captured by up-sampling and the aberration-free coherent transfer function serve as the inputs of the network, respectively. They are alternately updated and fed into the lighting update units with different angles (LUDA), as shown in Figure 4. A set of captured images $[eqn]$ and their corresponding wave vectors $[eqn]$ are taken as a sampling process, and in each sampling, all the samples with different angles are input into the model, and the model parameters are updated by using back-propagation. The expected results are generated through multiple sets of training phases, where the WFM module and the WFN module are separately shown in Figure 5a,b.

The pupil recovery module is specifically formulated as

$$[eqn]$$ $$[eqn]$$

where O(k) is the Fourier function of the sample, and $[eqn]$ and $[eqn]$ are the Fourier original aperture during the update of the INN_IWF network and the updated aperture, respectively. $[eqn]$ is a pre-upsampled sample image. $[eqn]$ represents the coherence transfer function of the objective lens, which is used to characterize the imaging quality of the diffraction-limited system under the condition of coherent illumination. The standard formula of pupil function CTF can be expressed as

$$[eqn]$$

where $[eqn]$ denotes the two-dimensional spatial coordinates of the Fourier domain, and NA denotes the numerical aperture, $[eqn]$ , where $[eqn]$ is the wavelength of the incident light.

Figure 4 shows the flowchart of LUDA. As the proposed network is defined in the complex domain, the samples and the coherence transfer function (CTF) are divided into real and imaginary parts, which are passed to the network as inputs to LUDA. The samples are shifted according to $[eqn]$ and then multiplied by the CTF to generate $[eqn]$ ,which is the spectrum before updating, Hence, Equation (5) can be rewritten as

$$[eqn]$$

where r and i represent the real and imaginary parts, respectively.

The traditional correction method cannot meet the requirements of complex aberration scenes. Therefore, this paper adds an optimization framework based on the traditional method, as shown in Figure 5a. The whole updating process can be represented by Equation (6). $[eqn]$ can be expressed by Equation (9).

$$[eqn]$$

where $[eqn]$ is the output of the WFM module in Figure 5a, represented by Equation (10), using the idea of the Wirtinger Flow algorithm [23]. As a technology for solving the phase retrieval problem, the Wirtinger Flow Algorithm [23] will transform the problem into a problem of finding the minimum value and serves as a general optimization framework that can reduce computational costs and effectively deal with noise. The spectrum $[eqn]$ before updating will be divided into two parts, one of which remains unchanged, and the other part is that $[eqn]$ is transformed by inverse Fourier transform and defined as $[eqn]$ , where $[eqn]$ is a linear sampling matrix, which is to be updated through operations such as phase subtraction, the dot product, etc. Then, the updated variable will undergo the Fourier transform again and be subtracted from $[eqn]$ to generate $[eqn]$ . The specific flow is shown in the WFM module of Figure 5a.

$$[eqn]$$

where ∆ is the custom gradient descent step size and ⨀ represents the dot product.

According to Equation (10), $[eqn]$ is gradient-updated to generate $[eqn]$ . $[eqn]$ enters the WFN module for phase conversion for calculating $[eqn]$ to obtain $[eqn]$ . Second, the amplitude of the simulated image in the WFN module is represented by the square root of the pre-sampled intensity image. As an intensity constraint, $[eqn]$ is multiplied by it. The network generates the updated spectrum $[eqn]$ according to the update process shown in Figure 5a,b. Since the spectra before and after the update have the same frequency, the whole network structure can be used to obtain the optimal result based on whether the difference between the spectra before and after the update is minimized. In this paper, the mean square error is used to calculate the minimum of the difference between the spectra before and after the update. The loss function is expressed as

$$[eqn]$$

2.3.2. Alternating Update Mechanism

After the above update process, the network outputs the updated samples and CTF. However, the samples and CTF have different properties when the network back propagates, and if the same gradient descent step size is used, the network will fail to converge to a perfect state. Therefore, an alternating update mechanism [24] is adopted to respectively control the gradient descent steps of the samples and CTF in this paper.

The updating process is divided into two parts, one of which aims to change the learning rate of the samples and control the gradient descent step size of the samples while keeping the CTF unchanged, and the other is to change the learning rate of the CTF to control the gradient descent step size of the CTF while keeping the samples unchanged, as identified by orange. Only after these two sections are completed will the network be able to converge to the optimal point and can better results be achieved for the samples and CTF.

2.3.3. Optical Aberration Processing Mechanism

The aberration function of the system is expressed in terms of Zernike polynomials, as shown in Equation (12), which can be used to describe the wavefront characteristics [25].

$$[eqn]$$

where ρ and θ are variables, $[eqn]$ is the expansion coefficient of different Zernike polynomials, and $[eqn]$ is different Zernike polynomials, which can be expressed as:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where m and n are positive integers with zeros, and $[eqn]$ are even numbers; n is the highest order ρ of the polynomial; m is the azimuth frequency; j is the order of the polynomial and is a function of n and m; and $[eqn]$ can be expressed as:

$$[eqn]$$

The CTF is always updated as a whole. The Zernike polynomials are applied to model the phase of the CTF in this paper, which, therefore, can be expressed as

$$[eqn]$$

where I in Equation (13) is the number of Zernike polynomials and $[eqn]$ is the coefficient of each Zernike polynomial.

The amplitude of the CTF remains updated as a whole, and the final form of the CTF modelling is expressed as

$$[eqn]$$

3. Experimental Results

3.1. Experimental System Setup

The equipment used for the experiments is shown in Section 3.1. A programmable controlled light source element LED and an illumination wavelength of 532 nm were used and placed 100 mm below the sample to provide illumination. In the sample collection process of the FPM device, the LED array is designed into a 15 × 15 LED rectangular area by programming. The rectangular region can be understood as a two-dimensional coordinate. The LED in the upper left corner of the coordinate starts to light up, and the remaining LED lights up in turn according to the coordinates, forming illumination at different angles. LED lights at different angles illuminate the samples placed on the stage. The FPM system used had a numerical aperture of 0.1 and was used to capture low-resolution sample images illuminated at different angles and record light intensity images using a CMOS camera with a pixel size of 3.45 µm. The results obtained by the INN_IWF were verified through both simulated and real datasets and then compared with those of other methods, such as those proposed by Jiang et al. [26].

Two metrics, namely, the Peak-Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), were used for evaluating the image quality. The Peak-Signal-to-Noise Ratio (PSNR) is an indicator commonly used to measure signal distortion. The larger the PSNR value, the better the image quality. In the field of image evaluation, the Peak-Signal-to-Noise ratio is calculated by the mean squared error (MSE):

$$[eqn]$$

The MSE is defined as

$$[eqn]$$

Among them, $[eqn]$ and $[eqn]$ represent the real image and the contrast image, respectively.

The Structural Similarity Index Measure (SSIM) is used to evaluate the image quality from the perspectives of brightness, contrast, and structure, which is in line with the intuitive effect observed by human vision, whose value falls in the range of 0~1:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ and $[eqn]$ , $[eqn]$ represent the mean and standard deviation of the two images, respectively; $[eqn]$ is the covariance of the two; and $[eqn]$ and $[eqn]$ are constant and equal.

3.2. Comparative Experiments with Simulated Datasets

The Cameraman and street map were used as the amplitude and phase images for the simulated dataset, as shown in Figure 6. The optical aberration is dominated by defocus aberration, which is caused by an uneven sample or inaccurate focusing. The experimental equipment, as described above, was used to generate 225 intensity images, from which the amplitude, phase, and CTF were reconstructed.

3.2.1. Correction Performance for Different Defocus Planes

Three defocus planes of 25 µm, 50 µm, and 75 µm were selected to verify the aberration correction performance of the method at different defocus planes (ranging from 25 µm to 75 µm). In this paper, Zernike polynomials were used to estimate the aberration, and the polynomial mode $[eqn]$ is about −1.44, corresponding to the defocus aberration of 50 µm. The first column is the low-resolution images with aberrations generated using the forward imaging model, the second and third columns are the images without aberration correction, and the fourth and fifth columns are the images after aberration correction using the INN_IWF network.

Figure 6 demonstrates the effect of aberration on the reconstructed results at different defocus planes. As can be seen from the figure, the effect of aberration on the final generated image became increasingly obvious with the increase in the amount of defocus. Compared with the image without aberration correction, the imaging effect after aberration correction using this method was improved, suggesting that the INN_IWF network can complete the correction of aberrations and maintain a good correction performance on different defocus planes.

In order to further verify the good aberration performance of the proposed method on different defocus planes, INNM [11] and EPRY [9] are used as comparison algorithms in this paper. Several experiments were carried out to compare the correction performance of the above three aberration correction methods on different defocus planes, and the PSNR and SSIM index values calculated by each experiment were averaged. As shown in Figure 7, the images constructed using the three methods were affected to some extent with the increase in the amount of defocus in different defocus planes. Among them, the EPRY method is most affected by the change in the defocus plane, while the method in this paper is least affected by the defocus plane, which can correct the aberration well and obtain the reconstructed image with richer image details. Table 1 is the image reconstruction index values of different methods on different defocus planes, among which the optimal results are marked in bold. In Table 1, the maximum and minimum values of the image reconstruction indexes calculated by many experiments are also shown. The fluctuation range of the maximum and minimum values in Table 1 is smaller than that of the other two methods. The purpose of the maximum and minimum values is to show the fluctuation range of the evaluation indexes of each method. It can be seen from the results shown in Table 1 that the EPRY method has a lower calculated evaluation index value than the other two methods because its correction performance is greatly affected by the change in the defocus plane. The method in this paper adds an optimization process to the network. Compared to the INNM method, it has a better performance and higher image evaluation index value. The above analysis shows that the method put forward in this paper consistently exhibited good aberration correction performances on different defocus planes.

3.2.2. Comparison of the Results of Different Methods on the Simulated Dataset

The results of this method were compared with those of INNM [11], EPRY [9], and the method proposed by Jiang et al. [26] on a simulated dataset under the condition that defocus aberration was used as the optical aberration, with a size of 50 µm. In addition, the PSNR and SSIM index values for each experimental result of the above methods were calculated and averaged, as shown in Figure 8 and Table 2. The results shown in Figure 8 show that the method in this paper can correct the aberration well. Compared to the other three methods, it has a higher image clarity and more image detail features. In Table 2, the optimal results are marked in bold. Table 2 shows the maximum and minimum values of the image reconstruction indexes calculated by Jiang et al.’s [26] method. The values of other methods are shown in Table 1. The results indicated that the results obtained by the method proposed in this paper were better than those obtained by the other three methods.

3.3. Comparative Experiments with a Real Dataset

3.3.1. Correction Performance for Different Defocus Planes

In order to verify that the proposed method still has a good correction performance in the face of complex aberration conditions, the device of Section 2.1 is used for sample collection. The numerical aperture of the system and the position of the LED array remain unchanged. The 15 × 15 LED illumination array irradiates the real cell image placed on the stage through the plane wave of different angles. The CMOS camera with a pixel size of 3.45 µm captures 225 real sample images with different angles of illumination and records the light intensity image. The intensity and phase images of the real samples are shown in Figure 9a,b.

Three defocus planes of 25 µm, 50 µm, and 75 µm were selected for comparison. The aberration correction results for different defocus planes are shown in Figure 10. The first column shows a low-resolution image with aberrations generated using the forward imaging model. The second and third columns are images without aberration correction. The fourth and fifth columns are images after aberration correction using the INN_IWF network.

Figure 10 shows the effect of aberrations on the reconstruction results of cell images at different defocusing planes, which shows that the effect of the aberration on the final reconstructed image became more and more pronounced with the increase in the defocus amount. Compared with the image without aberration correction, the imaging effect of the image corrected by the method proposed in this paper was improved, and the image texture features were retained to a large extent, implying that the INN_IWF network could not only achieve aberration correction but also maintain a good aberration correction performance in the case of severe aberration, so the reconstructed results retained more image detail features.

In order to further verify the good aberration correction performance of the method presented in this paper for cell images on different defocus planes, INNM [11] and EPRY [9] are used as comparison algorithms. The correction results of the above three aberration correction methods on different defocus planes are compared by multiple experimental results, and the PSNR and SSIM index values calculated by multiple experimental results are averaged. As shown in Figure 11, the defocus amount of different defocus planes gradually increased, which indicates that aberrations on the reconstruction results had a more and more obvious influence on the reconstruction results and that they would also have a certain degree of influence on the reconstruction image quality of the above three methods. The optimal results are in bold in Table 3. The maximum and minimum values of the image reconstruction indexes calculated by multiple experiments are also shown in Table 3. The fluctuation degree of the maximum and minimum values of the proposed method is the same as that of the INNM method, but the numerical value is better than that of the INNM method. As can be seen from the table, aberration had the greatest influence on the EPRY [9] method, and the proposed method and the INNM [11] method are less affected by aberrations. Table 3 also shows that the aberration correction effect of the method proposed on different defocus planes was better than that of the other two methods, with a higher value of the image reconstruction index. The above analysis shows that the proposed method maintains a good aberration correction performance for cell images, and the correction performance is not reduced in complex scenes while retaining image texture features.

3.3.2. Comparison of the Results of Different Methods on a Real Dataset

The dataset used in this subsection is four sets of cell images acquired under real experimental conditions, and the superiority of the method is verified through a comparison with other methods.

The results of multiple experiments of INN_IWF, INNM [11], EPRY [9], and the method proposed by Jiang et al. [26] in real datasets are compared, as shown in Figure 12. Table 4 is the average value of the image reconstruction index of the above four methods in PSNR and SSIM. Due to the limitation of the table size in Table 4, the maximum and minimum values of multiple sets of real image reconstruction indexes are shown in Table 5. It can be seen from the results that the method in this paper is better than the other methods. The optimal results are in bold. It can be seen from Figure 12 and Table 4 that the image clarity obtained by the INN_IWF was improved when compared with that of the methods proposed by Jiang et al. and EPRY in these four groups of experiments, with more image details. As can been from the reconstruction indexes in Table 4, the reconstruction index value of the method proposed in this paper was higher. The results of the first two groups of experiments were similar to those of the INNM method, while the results of the latter two groups show that the correction performance of the proposed method is better than that of the INNM method. The image reconstruction index values of the two methods in Table 4 show that the INNM is suboptimal. In summary, the method in this paper had a better aberration correction performance in real datasets and was able to obtain better reconstruction results.

4. Conclusions

This paper proposes an aberration correction method based on the improved Wirtinger Flow algorithm under the Tensorflow framework. This method simulates the forward imaging process and improves the Wirtinger Flow algorithm introduced into the model, retains the central idea, simplifies the calculation process, and improves the performance of the aberration correction of the network. The alternating update mechanism (AU) updates the sample and the coherent transfer function in batches to obtain better results. Zernike polynomials can estimate aberrations with high precision. The simulation and experimental results show that the INN_IWF network demonstrates a better performance in correcting aberrations while obtaining richer texture details of reconstructed images, proving that the proposed method is superior on different defocus planes, effectively avoiding a low correction accuracy and poor correction performance under complex aberration conditions while retaining more image texture features when compared to traditional algorithms.