Structured illumination microscopy based on fiber devices

TL;DR

This paper introduces a compact fiber-based structured illumination microscopy system that uses fiber couplers and MEMS switches to produce illumination patterns, enabling super-resolution imaging through advanced reconstruction algorithms.

Contribution

A novel fiber device setup for structured illumination microscopy that simplifies the system and enhances stability and speed of pattern switching.

Findings

Achieved super-resolution imaging with 4-frame data.

Demonstrated stable and high-speed illumination switching.

Improved image quality by removing out-focus components.

Abstract

We present a simple and compact approach of structured illumination microscopy by using three $2\times2$ fiber couplers and one $1\times4$ MEMS optics switch. One uniform and three fringe illumination patterns were produced by placing seven output fiber tips at the conjugate Fourier plane of the illumination path. Stable and relatively high-speed illumination switching was achieved by the optics switch. Super-resolution and optical sectioned information was reconstructed from 4-frame data by using algorithms based on a joint Richardson-Lucy deconvolution method and a Hilbert transform method. By directly removing the out-focus components from the raw images provides better imaging results.

Equations

$$S_{SR}=\underset{\hat{S}}{\min }[\sum _{n=1}^N(M_n-(I_n\times \hat{S})\otimes h{)}^2],N\ge 3$$

$$\hat{M}=h\otimes (S_i\times I)+\mathcal{N},$$

$$r=M/\hat{M},$$

$${\hat{S}}_{i+1}={\hat{S}}_i\times [h\otimes (r\times I)]/[h\otimes (ones\times I)],$$

$$M_l(r)=S_{O{F}_l}(r)+S_{I{F}_l}(r)\times cos(2\pi kr+\varphi ),$$

$$M_{IF}(r)=M(r)-R(r).$$

$$S_{IF}=\left|M_{IF}+i\mathcal{H}\right|.$$

$$\mathcal{H}=\left|{\mathcal{F}}^{-1}[\mathcal{F}(M_{IF})\mathcal{S}]\right|,$$

$$S_{OS}=\frac{1}{N}\sum _{n=1}^N{S}_{I{F}_n}.$$

$$S={\mathcal{F}}^{-1}[\beta \mathcal{F}(S_{OS}{\mathrm{Mask}}_{\alpha })+(1-\beta )\mathcal{F}(S_{SR})(1-{\mathrm{Mask}}_{\alpha })],$$

$$M_{I{F}_n}=M_n-\delta {\mathcal{F}}^{-1}[\mathcal{F}(M_4-S_{I{F}_n}){\mathrm{Mask}}_{\gamma }]$$

$${\mathrm{Mask}}_{\alpha }=\{\begin{array}{ll}1& u^2+v^2\le {\alpha }^2\\ 0& u^2+v^2>{\alpha }^2\end{array}$$

$$I(r)=0.5(1+cos(2\pi \vec{k}r)),$$

$$M(x,y,z_0)=\mathcal{P}((\sum ^Z(O(x,y,z)\times I(x,y))\otimes PSF(x,y,|z-z_0|){)}_{\downarrow 2}),$$

Full text

††journal: Optics Communications

1 Introduction

Structured illumination microscopy (SIM) has been an efficient way to enhance the resolution of the fluorescent microscopy [1]. Compared to confocal microscopy, SIM has the advantages of being a wide-field imaging system, and has the same optical section capability as well [2, 3]. To achieve both super-resolution (SR) and optical sectioning (OS), SIM requires raw SI images with three or five phase-shifts at three orientations. To fulfill this requirements, structured illumination (SI) methods with phase shift have been achieved by using gratings [4], SLMs [5, 6], integrated optics chips [7, 8], galvanometers [9], and other devices [10, 11].

Actually, the desire of phase shift makes it complex and expensive to build a SIM system, which needs those devices to modulate the phase accurately. Recently, several SR algorithms that require only four raw images (one wide-field and three SI images) have been proposed [12, 13, 14, 15]. Meanwhile, it is also possible to achieve OS from non-phase-shift raw images [16, 17, 18]. On one hand, those methods increase the frame rate by reducing the number of raw images; on the other hand, they do not require any phase shift of the SI pattern, which could be used to simplify the optical setup. In our previous work [19], we placed a $2\times 2$ fiber coupler’s output tips at the conjugate Fourier plane of the illumination path to produce fringe patterns at the sample plane. By rotating the fiber mount, SI images at multiple orientations ($\geq 3$) without any phase shift were captured and SR result was reconstructed by a joint Richardson-Lucy (jRL) deconvolution method [20]. Although this method is simple and low-cost, both of the quality of the SI pattern and the imaging speed of system are limited by the mechanical rotation of the fiber tips.

In this paper, we proposed an improved SIM system using three $2\times 2$ fiber couplers and a $1\times 4$ MEMS optics switch with no moving part. We modified a commercial microscope (IX-83, Olympus) by placing seven fiber tips at the conjugate Fourier plane of the illumination path, which could provide three SI and one uniform illumination at the sample plane. The use of the optics switch significantly increased the speed of SI pattern generation and the stability of the system, comparing to our previous work [19]. Both SR and OS images were reconstruct by using a jRL deconvolution method and a single-slice optical section algorithm based on Hilbert transform. In addition, we presented two different strategies to merge the SR and OS components, and it shows that by directly removing out-focus information from each raw images will yield a better imaging result.

2 Method

Fig. 1 shows a schematic diagram of our optical setup. A 532 nm laser source (LC-651A Laser Combiner, ColdSpring) is directly coupled into a $1\times 4$ MEMS optics switch (Ziguan, Sichuan China), SW. Three of switch’s output channels (ch1-ch3) are connected to three $2\times 2$ single mode fused fiber optic couplers (TN532R5F2, THORLABS), FC1-FC3, with 50:50 coupling ratio, whose output tips are mounted on a 3D printed fiber mount, M, in concentric arrangement. And the remained channel (ch4) is directly mounted on the center of the fiber mount, while the fiber mount is placed at the conjugate Fourier plane of the illumination path. The layout of fiber tips’ position is shown in the upper panel of the Fig. 1. All the fiber tips are placed in parallel and they directly outputted the light from 2.5 $\mu m$ cores which could be considered as point light sources. Two lenses (L1-L2) are used to collect and projected the light into the back aperture of the microscope. The z-scan moving stage we used could provided 0.01 $\mu m$ resolution in focusing control with maximum speed of 5 mm/sec. A sCMOS camera (ORCA-Flash 4.0 V2, HAMAMATSU) is used to capture the fluorescent signal. The tube lens of the system, L5, is a 180 mm lens.

In this setup, each of the fiber couplers could split the incident light into two beams which could produce an interference pattern at the sample plane. Meanwhile, the light outputted from the center tip could provide a uniform illumination. In other words, 4 raw images (three SI images and one wild-field image) can be captured by switching the channel of the optics switch. As we previous proposed [19], the frequency of the illumination patterns could be modulated by using different focal length combination of L1 and L2 or using different fiber mount with different distance between fiber tips. It should be mentioned that L3 (125 mm) and L4 (180 mm) are build-in lenses of the microscope which is a 4F system, while L1 and L2 are not. In practice, we places L1 and L2 as close as possible to get a higher power efficiency.

As we mentioned above, in our Method, we only captured one wide-field and three SI images, which did not contain any phase shift. Here we employed two methods to recover both SR and OS information from raw images. As for SR part, a jRL deconvolution [20, 21] was used, which reformulated the matrix inversion problem as a minimization problem [22]:

[TABLE]

where $S_{SR}$ is the estimated high-resolution image of sample’s fluorescence, $M_{n}$ is the $n^{th}$ captured image, $I_{n}$ is the $n^{th}$ illumination pattern, $\otimes$ is the convolution operator, $h$ is the point spread function (PSF). This minimization problem can be solved based on iterations. For each iteration in jRL deconvolution:

[TABLE]

[TABLE]

[TABLE]

where $S_{i}$ is an estimate of the real fluorescence distribution of the sample in $i^{th}$ iteration, and $ones$ is a matrix of $1s$ with the same dimensions as the $M$. By inputting all the captured images, $M_{n}$, and their illumination profile, $I_{n}$, iterations will yield high resolution images. It should be mentioned that jRL deconvolution method has no OS capacity. Fine details about the jRL deconvolution algorithm could be found in [20, 21]. As for OS part, we employed a single-slice optical section algorithm based on Hilbert transform method [16] to demodulate the in-focus information from each SI images. Each structured illuminated image can be considered as the sum of modulated in-focus signal and unmodulated out-focus signal:

[TABLE]

where $S_{OF_{l}}(r)$ and $S_{IF_{l}}$ separately represents to the in- and out-focus components. A notch filter (filtering out the $\pm 1$ peaks in Fourier domain) and a low-pass filter was used to estimate unmodulated component of each SI images, $R$. Hence, the modulated in-focus component can be separated by subtracting $R$:

[TABLE]

Then, the in-focus image can be demodulated using Hilbert transform [21]:

[TABLE]

Here $\mathcal{H}$ is the two-dimensional Hilbert transform of the $M_{IF}$,

[TABLE]

where $\mathcal{F}$ and $\mathcal{F}^{-1}$ represent to Fourier transform and inverse Fourier transform, and $\mathcal{S}$ is the spiral function in Fourier domain, $\mathcal{S}(u,v)=\frac{u+iv}{\sqrt{u^{2}+v^{2}}}$. Here $f$ is the spatial frequency. By adopting this process to all the SI raw images and taking an average of them, the sectioned image, $S_{OS}$, could be calculated:

[TABLE]

It should be noted that an accurate estimation of the unmodulated signal, $R$, is very important and also difficult during the demodulation using Hilbert transform [23].

Finally, the high-resolution SR and the low-resolution OS information should be merged in to one final image. We employed two strategies (Strategy 1 and Strategy 2). The different performance of these two strategies will be discussed in more details in the experiment.

Strategy 1: We first estimate the SR image ($S_{SR}$) from measured raw images ($M_{n}$) using eq. 1 and the OS image ($S_{OS}$) by using eq. 9. Then we merged $S_{SR}$ and $S_{OS}$ images in Fourier domain [24, 25] to get the final result ($S$):

[TABLE]

where $\mathrm{Mask}_{\alpha}$ is a low-pass filtering mask in Fourier domain, $\alpha$ represents to the radius of passband, and $\beta$ is a positive weight coefficient.

Strategy 2: We first calculate the in-focus information ($S_{IF_{n}}$) of each raw images ($M_{n}$) by using eq. 7. Then the in-focus information ($M_{IF_{n}}$) could be estimated by subtracting their out-focus component:

[TABLE]

where $\mathrm{Mask}_{\gamma}$ is a low-pass filtering mask in Fourier domain, $\gamma$ represents to the radius of passband, $\delta$ is a positive weight coefficient, and $M_{4}$ is the wide-field image. Final result ($S$) is reconstructed from $M_{IF_{n}}$ by using eq. 1.

It should be mentioned that in the image processing we set the filter mask to be an ideal low-pass filter,

[TABLE]

.

3 Numerical Simulation

To verify the performances of those methods we mentioned above, we used a 3-dimention object ($256\times 256\times 256$) shown in Figure 2 (a) to simulate the imaging process. In the simulation, the excited wavelength was 570 $nm$; the numerical aperture of the imaging system was 1; the size of each pixel was 50 $nm$. Then the illumination pattern was set as

[TABLE]

where $\vec{k}_{n}$ represents to the wave vector, and we set $|\vec{k}|=0.6f_{cut-off}$ which is close to the setting of our optical setup. One of the illumination pattern is shown in Figure 2 (b). The PSF we used in the simulation was generated using Born-and-Wolf model [26], as shown in Figure 2 (c). Mathematically, the acquirement of the raw SI images could be expressed as

[TABLE]

where $M(x,y,z_{0})$ donates the measured image at $z=z_{0}$ plane, $O(x,y,z)$ represents to the 3D object, $\downarrow 2$ donates a downsampling by a factor of 2, and $\mathcal{P}$ represents to the Poisson distribution. In the simulation, one uniform-illuminated image (Figure 2(f)) and three SI images in three orientations (Figure 2(e)) at $z=0$ plane were acquired. Figure 2 (d) shows the ground-truth image of the object at z = 0 plane.

Figure 2(g-k) shows the simulation results reconstructed by those methods we mentioned above: (g) deconvolved image; (h) deconvolved sectioned image using Hilbert transform method; (i) super-resolution image using jRL deconvolution method; (j) reconstructed image of Strategy 1; (k) reconstructed image of Strategy 2. In the reconstruction, we employed 50 jRL iterations in all the deconvolution processing, and we set $\alpha=0.3f_{cut-off}$, $\beta=0.5$, $\gamma=0.3f_{cut-off}$ and $\delta=1$. The red circles in the figure are corresponding to the resolution limit. Sub-diffraction details (within the red circle) were distinguished in Figure 2(i-k), but not in Figure 2(f-h), which shows that super-resolution results were achieved by using jRL deconvolution method. As shown in Figure 2(h,j,k), out-focus information was removed in the reconstructed results. It should mentioned that the sectioned image, shown in Figure 2(h), reconstructed by Hilbert transform method, contains some low-frequency artifacts.

In order to further compare the different super-resolution and optical sectioning performances of those reconstruction strategies, we analyzed two line intensity profiles (high-frequency and low-frequency area) pointed out in Figure 2(f) by yellow and blue line. As shown in Figure 2(l) and (n), both Strategy 1 and 2 could achieve good super-resolution and optical sectioning performances. In high-frequency region (yellow line), the fine details in the result of Strategy 1 were influenced by the low-frequency artifacts from the sectioned image (pointed out by red arrows), though they have a similar resolution comparing to the result of Strategy 2. In relatively lower-frequency region (blue line), the result of Strategy 2 has a clear background, but its structures are too thin comparing to the ground-truth image, which lost some information of the real object.

The settings of those variables’ value in eq. 10 ($\alpha$ and $\beta$) and eq. 11 ($\gamma$ and $\delta$) make significant influence to the final imaging result; there is a trade-off in the combination of the super-resolution and optical sectioning. Actually, these variables can be divided into two categories according to their functions: one is to match the spatial frequency range ($\alpha$ and $\gamma$); and another is to match the intensity ($\beta$ and $\delta$). It should mentioned that, in the reconstruction, we assumed that all the out-focus information was low-frequency, which means that only the low-frequency part of the image needs to be sectioned. In Strategy 1, the value of $\alpha$ ($0\leq\alpha\leq f_{cut-off}$) determined the frequency range of the merging images; the low-frequency component ($f<\alpha$) of the OS image and the high-frequency component ($f>\alpha$) of the SR image are selected. And these two components are finally added together by an intensity adjustment factor $\beta$ ($0\leq\beta\leq 1$). As shown in Figure 3 (a), if $\alpha$ is too small, there will be little optical sectioning effect; in contrast, is $\alpha$ is too large, the final result will contain the low-frequency artifacts produced by the single-slice optical section processing. The influence of setting different value of $\beta$ is shown in Figure 3 (b). Meanwhile, in Strategy 2, $\gamma$ ($0\leq\gamma\leq f_{cut-off}$) plays a similar role as $\alpha$ in Strategy 1, which limited the frequency range of the out-focus component as shown in eq. 11. And $\delta$ ($0\leq\delta\leq 1$) is used to adjust the intensity of the out-focus information in order to obtain a good estimation of in-focus component of each raw images. Those influences could be seen in Figure 3 (c) and (d). However, the setting of these variables depends on the actual situation.

In all, Strategy 2 that directly removes the out-focus components from the raw images provides better performance.

4 Experiment

In the experiment, we set the distance of each fiber tip pair to 6 mm, and focal length of L1 and L2 were 200 mm and 100 mm. Considering the build-in lenses, L3 (125 mm) and L4 (180 mm), those point light sources (outputted by each fiber pairs) were projected on the back aperture of the OL (UPlanFLN, 60$\times$/1.25, Olympus) with a distance of 4.32 mm (the diameter of the back aperture of the OL was 7.5 mm), which means the SI patterns were set to $57.6\%$ of the maximum spatial frequency of the excitation light. The numerical aperture of the OL can be adjusted from 0.65 to 1.25. The light source used in the experiment was a 532 nm laser source with a maximum power of 100 mW, and a Cy3 filter cube ($\lambda_{ex}=513\sim 556nm$ ; $\lambda_{em}=570\sim 613nm$) was used to filter the light.

We first used 0.04 $\mu m$ fluorescent beads (F10720, Red, Thermo Fisher Scientific) to calibrate our microscope using the uniform illumination channel of the optics switch (ch4). By averaging 22 separated point spots, the point spread function (PSF) could be obtained by a first order Gaussian fitting, whose FWHM (full width at half maximum) is 341 nm in the experiment, as shown in Figure 4; the resolution limit of this imaging system is 341 nm. As shown in Figure 4(b), The spatial frequency of the illumination pattern is about $0.6f_{cut_{o}ff}$, which shows that the resolution enhancement factor of our setup is about 1.6 in theory.

Then we imaged Rhodamine Phalloidin ($\lambda_{ex}=540\sim 545nm$ , $\lambda_{em}=570\sim 573nm$) labeled CaSki cells to verify both of SR and OS performance of our method. 27 layers with a step of 0.1 $\mu m$ in vertical axis were imaged; four raw images were captured per layer. The exposure time was set to 300 ms, and it took about 10 ms for the optics switch to switch illumination channel; The acquisition time for each layer was about $1.230s$, and it took about $34.510s$ to complete the whole z-scanning. Then we employed the image reconstruction methods described above, and both super-resolution and optical sectioned were achieved. Fig. 5 (a) shows a $1024\times 1024$ pixels region. A wide-field image is shown in Fig. 5 (a1), while Fig. 5 (a4) and (a5) show the reconstructed results by using Strategy 1 and 2. To be more convincing, we also performed the Hilbert transform method and jRL deconvolution method individually, and the results are shown in Fig. 5 (a2) and (a3). In the image reconstruction, all jRL deconvolution processing was employed 50 iterations, and the variables were set as: $\alpha=0.1f_{cut-off}$, $\beta=0.5$, $\gamma=0.1f_{cut-off}$ and $\delta=1$, where $f_{cut-off}$ represents to the cut-off frequency of the microscope. It takes about $36.157s$ for Strategy 1 and $36.682s$ for Strategy 2 to calculate one slice image ($1024\times 1024$ pixels) by using a PC (Intel Core i7-6700 3.40GHz processor and 16GB RAM) with Matlab software (R2018b) under Windows 7 (SP1) x64 operating system.

As the line fluorescent distribution profile shown in Fig. 5 (b), sub-diffraction resolution of $\sim 250nm$ ($<341nm$) was achieved by using jRL deconvolution; the images reconstructed by employing only jRL deconvolution method, Strategy 1 and 2 had been resolution enhanced. To further verify the optical section performance, Fig. 5 (c1) - (c5) show a vertical cross section images of the sample. Low-resolution out-focus information was removed by using Hilbert transform method, but it lost lots of fine detail of the sample, as shown in Fig. 5 (a2) and (c2). Although fine detail was recover in Fig. 5 (a3) and (c3), jRL deconvolution method cannot achieve optical sectioning. Meanwhile, as shown in Fig. 5 (c4) and (c5) pointed out by blue arrows, the structure is thin without any trailing in vertical axis, which indicated that both Strategy 1 and Strategy 2 could obtain super-resolution and optical sectioned images. It also shows that more out-focus information was removed in Fig. 5 (c5) than in (c4), while they have similar resolution.

5 Discussion

The main purpose of this work is to find a simple and low-cost way to build a stable SIM setup with acceptable imaging speed and similar imaging performance (both super-resolution and optical sectioning) comparing to other SIM setups. For optical setup, there are two main parts of the SI pattern generation method. One is the generation of two coherent point light sources. We used three fiber coupler to split one laser source into point-light-sources pairs (the output tips of FCs), which could produce interference patterns at the sample plane in three orientations. However, the single mode fused fiber optics coupler we used had a narrow band (532 $\pm$ 15 nm), which limited its application in multi-color imaging. In our experience, a planar lightwave circuit (PLC) splitter with multiple operating wavelengths will be a preferred choice. And another part is the switching of the illumination modes, which was achieved using a $1\times 4$ optics switch whose switching time about $10ms$; the max SI generation speed is about $100Hz$. Compared with our previous work [19], the stability and the imaging speed was significantly improved in this approach. As shown in Fig. 6, we analyzed the different pattern modulation performance of both optics switch and mechanical rotation by imaging some fluorescent dirt from a highlighter pen. The frame rate of the camera was set to 200 Hz, and the time period of each SI channels was set to 200 ms. As shown, for mechanical rotation, there was a rapid phase shifting in random after rotation operation due to the inertia of the fibers in free space, which caused artifacts in image reconstruction, and it took about 300 ms to stabilize the pattern; for optics switch, the phase was barely shifted in each orientations, and the orientation switching was fast, which only took about 10 ms. Moreover, the phase was identical in each orientations by using optics switch, which reflected the stability improvement of the system in the other way. However, the switching speed of the MEMS optics switch is not quite enough for ultra-high-speed fluorescent imaging. Integrated optics devices seem to be a preferred solution of both small volume and high speed of channel switching in the future, and some related works have been already proposed recently [8, 7].

Nowadays, SLMs are the most widely used devices to achieve accurate and high-speed structured illumination by loading different patterns on their chips. Like grating, SLMs will diffract the incident laser light into multiple beams, which means they need a filter system to achieve two-beam of three-beam interference. In addition, diffraction also causes low light efficiency. Others devices like galvanometers or DOEs, they need to align multiple optical elements accurately. Compared with those methods mentioned above, our method based on fiber devices made some improvements that by putting outputting fiber tips at the conjugate Fourier plane shorten the illumination path and also increases the light efficiency [19]. Meanwhile, it is much simple and low-cost to build, while most of the fiber devices can be flexiblely assembled in small volume. As a result, our method is more likely to be commercialized.

As for the image processing, 4-frame SIM data (three SI and one wide-field raw images) meets the minimum data acquirement to recover the real super-resolution image of the sample fluorescent distribution [22]. And it increased the frame rate in some ways, compared to other method based on 9- or 15-frame SI raw images. Meanwhile, using a single-slice algorithm rather than other optical section algorithms based on $>1$ raw images [2, 27] could avoid the influence of the uneven excitation intensity of different illumination channels, which could provide better sectioning performance.

In the combination of SR and OS, we employed two different strategies to combine the results of super-resolution and optical sectioning. Those two strategies have their own advantages and disadvantages. In Strategy 1, the jRL deconvolution method and Hilbert transform method are performed separately, and then these two components were merged in Fourier domain. It is suit for all kinds of SIM methods where the super-resolution and optical sectioning could individually performed. However, the “out-focus” information will lower the modulation factor in pattern estimation. In Strategy 2, the in-focus components of the raw SI images are demodulated by using Hilbert transform method first, and then we performed jRL deconvolution method with these “in-focus” data to reconstruct final result. It avoids to jRL deconvolve the out-focus component with illumination patterns. However, it relies on a good estimation of uniform illuminated image, which makes it only suit for those SIM methods that could provide a wide-field image or yield a uniform-illuminated image by summing the SI images.

6 Conclusion

In summary, we proposed a compact and low-cost approach of generating structured illumination by using three fiber optics couplers and one MEMS optics switch. By placing seven fiber tips at the conjugate Fourier plane of the illumination path, interference patterns were projected on the sample plane. The switching of the illumination channels using the optics switch is stable and relatively high-speed. Then super-resolution and sectioned image was reconstructed from 4 raw images by using algorithms combining jRL deconvolution method and Hilbert transform method. Moreover, we shows that better imaging result could be obtained by directly removing the out-focus components from the raw images. Our method provides a simple way to build a SIM setup under 4-frame SIM framework, which offers a similar imaging performance and acceptable imaging speed comparing to the conventional SIM setups and has its potential of commercialization in the future.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China [2018YFF0214900], National Natural Science Foundation of China [21804126], and the Instrument Developing Project of the Chinese Academy of Sciences [YZ201402].