A Super-Condenser for Labelfree Nanoscopy

TL;DR

This paper introduces a novel labelfree nanoscopy technique combining Fourier ptychography with waveguide microscopy, achieving high-resolution imaging by using a super-condenser with inclined darkfield illumination and advanced phase retrieval methods.

Contribution

It presents a new super-condenser design utilizing Si3N4 waveguides for enhanced coherent illumination in Fourier ptychography, enabling larger imaging areas and improved phase retrieval.

Findings

Successful in silico validation of the method

In vitro experiments demonstrate high-resolution imaging

Enhanced phase retrieval via multiplexed waveguide illumination

Abstract

Labelfree nanoscopy encompasses optical imaging with resolution in the 100 nm range using visible wavelengths. Here, we present a labelfree nanoscopy method that combines Fourier ptychography with waveguide microscopy to realize a 'super-condenser' featuring maximally inclined coherent darkfield illumination with artificially stretched wave vectors due to large refractive indices of the employed Si$_3$N$_4$ waveguide material. We produce the required coherent plane wave illumination for Fourier ptychography over imaging areas 400 $\mathrm{\mu}$m$^2$ in size via adiabatically tapered single-mode waveguides and tackle the overlap constraints of the Fourier ptychography phase retrieval algorithm two-fold: firstly, the directionality of the illumination wave vector is changed sequentially via a multiplexed input structure of the waveguide chip layout and secondly, the wave vector modulus is shortend via step-wise increases of the illumination light wavelength over the visible spectrum. We validate the method via in silico and in vitro experiments and provide details on the underlying image formation theory as well as the reconstruction algorithm.

Equations

$$\Delta x=\frac{\lambda }{N{A}_c+N{A}_o}$$

$$i_{k_0,k_c}(x)={\left|\left[s(x)\times \text{exp}\left(-\mathrm{i}{k}_0\times x\right)\right]\otimes h_c(x)\right|}^2.$$

$$a_{k_0,k_c}(x)=\Re \left\{\sqrt{{\mathfrak{F}}^{-}\left\{\mathfrak{F}\left\{i_{k_0,k_c}(x)-b\right\}\times H_c\left(\frac{k}{2}\right)\right\}}\right\}.$$

$$t_{k_0,k_c}^j(x)={\mathfrak{F}}^{-}\left\{F^j(k-k_0)\times H_c\right\}.$$

$$t_{k_0,k_c}^{j+1}(x)=a_{k_0,k_c}(x)\times \text{exp}\left(\mathrm{i}\Phi (t^j(x))\right).$$

$$F^{j+1}(k)=F^j(k)\times \left(1-H_c(k-k_0))\right)+H_c(k-k_0))\times T_{k_0,k_c}^{j+1}(k-k_0).$$

$$f(x)={\mathfrak{F}}^{-}\left\{apo(pad(F(k)))\right\}.$$

$$\Delta x_{TiO}=\frac{405\text{nm}}{(1.49+2.66)}=97.6\text{nm}.$$

Full text

A Super-Condenser for Labelfree Nanoscopy

Florian Ströhl

Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway

Department of Chemical Engineering and Biotechnology, University of Cambridge, CB3 0AS Cambridge, UK

florian.strohl@uit.no

Ida S. Opstad

Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway

Jean-Claude Tinguely

Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway

Firehun T. Dullo

Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway

Ioanna Mela

Department of Chemical Engineering and Biotechnology, University of Cambridge, CB3 0AS Cambridge, UK

Johannes W.M. Osterrieth

Department of Chemical Engineering and Biotechnology, University of Cambridge, CB3 0AS Cambridge, UK

Balpreet S. Ahluwalia

Department of Physics and Technology, UiT The Arctic University of Norway, NO-9037 Tromsø, Norway

Clemens F. Kaminski

Department of Chemical Engineering and Biotechnology, University of Cambridge, CB3 0AS Cambridge, UK

Abstract

Labelfree nanoscopy encompasses optical imaging with resolution in the 100 nm range using visible wavelengths. Here, we present a labelfree nanoscopy method that combines Fourier ptychography with waveguide microscopy to realize a super-condenser featuring maximally inclined coherent darkfield illumination with artificially stretched wave vectors due to large refractive indices of the employed Si3N4 waveguide material. We produce the required coherent plane wave illumination for Fourier ptychography over imaging areas 400 µm2 in size via adiabatically tapered single-mode waveguides and tackle the overlap constraints of the Fourier ptychography phase retrieval algorithm two-fold: firstly, the directionality of the illumination wave vector is changed sequentially via a multiplexed input structure of the waveguide chip layout and secondly, the wave vector modulus is shortend via step-wise increases of the illumination light wavelength over the visible spectrum. We validate the method via in silico and in vitro experiments and provide details on the underlying image formation theory as well as the reconstruction algorithm.

1 Introduction

Conventional nanoscopy, optical microscopy with resolution below 100 nm, is based on fluorescence[1]. Often listed advantages of nanoscopy, especially in comparison to electron microscopy, are the simple sample preparation, live-cell compatibility, and molecular specificity. Though live-cell compatible, the introduction of fluorescent labels onto the molecular structures of interest are in living cells likely to cause both functional and structural aberrations, potentially leading to false conclusions, and is also associated with problems like photobleaching and phototoxicity, variable label specificity, imaging- and image reconstruction-related artifacts, and lengthy optimization protocols [2, 3]. The advantage of label specificity also has its downside of excluding (ultra)-structural context of the specifically labeled structure, although this can be alleviated to some degree via multi-channel labeling. Synergistic approaches combining the advantages of label specificity from conventional nanoscopy together with ultra-structural context obtained via label-free nanoscopy, could bring many new insights about cellular functions, especially as (contrary to correlative light and electron microscopy[4]), label-free (optical) nanoscopy has the potential of also being applied to living cells and cellular systems. Excluding methods that have not gone beyond the proof-of-principle stage like hyperlensing[5] or super-oscillation microscopy[6], suitable labelfree methods that have the potential to provide nanoscopic resolution can be sorted broadly into four groups: (1) Autofluorescence probed with conventional nanoscopy. Although certain critical fluorophore properties that are required for ultra-high resolution nanoscopy like photo-switching[7] are normally not present in intrinsically fluorescent samples, structured illumination microscopy has been shown to resolve features in the 150 nm regime in unlabeled retinal tissue[8]. Note that intrinsic fluorescence is a property present only in some but not all samples. (2) Nearfield scanning optical microscopy[9], which rasters a sample with an effective resolution below 100 nm. Akin to electron microscopy, this scanning optical approach has a low through-put and is challenging to combine with fluorescence-based nanoscopy. (3) Deep ultra-violet microscopy - a theoretically simple approach as resolution scales linearly with employed imaging wavelength. However, the limited availability and performance of optical components in this spectral range as well as the high phototoxicity associated with ultraviolet radiation offset the benefits gained by wavelengths below 400 nm illumination. (4) Fourier ptychography, FP[10], a technique specifically developed for improving digital pathology[11]. In FP the sample is illuminated and imaged sequentially with plane waves from a multitude of directions that densely sample the illumination condenser numerical aperture ($NA_{c}$). The generated set of images is then synthesized into a super-resolved amplitude and phase image of resolution $\Delta x$ given by

[TABLE]

via a dedicated phase retrieval algorithm[10]. It is well known that in conventional microscopy the condenser $NA_{c}$ used for illumination should be matched to (or even slightly below) the objective $NA_{o}$ [12], resulting in an effective maximal resolution of $\Delta x=\frac{\lambda}{2NA_{o}}$, the Abbe resolution limit [12]. Crucially in FP, a numerical aperture $NA_{c}$ of the condenser can be larger than the objective’s $NA_{o}$ in order to increase resolution with respect to the detection objective.

2 Theory

As the illumination in FP is coherent, the complex field (amplitude and phase) of the sample is probed rather than its intensity as in incoherent imaging. Even though the effective aperture of a coherent microscope is half the size of an incoherent one’s, plane wave illumination at oblique angles re-positions the sample’s field in the aperture, thus giving access to finer details as visualized in Figure 1a-c. Further, because the down-modulation of sample spatial frequency information with the illumination’s spatial frequency occurs before being low-pass filtered by the objective aperture, access to information beyond $2NA_{o}$ is possible given high lateral spatial frequency of the illumination at greater angles than conventionally associated with $NA_{o}$. To extract those finer details, multiple images are acquired sequentially using illumination angles spanning the entire condenser $NA_{c}$, and combined into a super-resolved image computationally (see Figure 1d). Despite its potential, an extended condenser $NA_{c}$ has so far almost exclusively been used to increase the space-bandwidth product [10] rather than performing nanoscopy. This is because, assuming the highest $NA$ available for both illumination and detection, the resolution caps at the incoherent resolution limit and is hence in the order of 200 nm. To resolve nanoscopic structures using FP, a super-condenser allowing illumination with spatial frequencies exceeding those offered by the best immersion objectives is necessary. In the following, we show how such a super-condenser can be implemented in the form of a photonic waveguide chip in conjunction with multi-spectral illumination. We show simulation results of the proposed method and perform proof-of-concept imaging of sub-diffraction-limit sized metal-organic framework (MOF) clusters. Figure 1e outlines the fundamental mechanism of the proposed super-condenser, which aims to optimize the magnitude of the lateral illumination wave vector components and to provide as dense coverage of the virtual condenser pupil as possible. To achieve largest lateral wave vector components, the illumination administered to the sample via photonic waveguides is intrinsically orthogonal to the detection objective and thus allows to maximize the wave vector magnitude geometrically. Furthermore, akin to microscopy with immersion media, the illumination wave vector is stretched by a factor determined by the refractive index of the waveguide material. Thus, by apt choice of material, a further tremendous increase in wave vector magnitude and image resolution can be achieved. To illustrate, a conventional fluorescence microscope imaging GFP ($\lambda_{ex/em}=488/512$ nm) with a high-performance 0.95 NA air objective offers a maximal resolution of $\lambda_{em}/2NA_{o}\approx 270$ nm. The same sample imaged with the proposed super-condenser featuring Si3N4 waveguides with refractive index n $\approx 2.08$ yields a nominal resolution of $\lambda_{ex}/(\text{n}+NA_{o})\approx 160$ nm. Furthermore, it should be considered that incoherent microscopy techniques (like fluorescence or brightfield) have a strongly damped optical transfer efficiency for higher spatial frequencies, whereas the effective transfer function produced by Fourier ptychography has close to unity transmission strength, thus obtaining greatly enhanced contrast for finer structural details (as visualised in Figure 1).

3 Reconstruction Algorithm

As our technique is based on Fourier ptychography it uses a phase retrieval algorithm[10], which was slightly modified and is depicted in the box of Figure 2. The algorithm aims to invert the imaging pipeline and thus requires a detailed description of the image formation. Let the raw images be denoted as $i_{k_{0},k_{c}}(x)$, with $k_{0}$ being the illumination wave vectors’ lateral component, and $k_{c}$ the coherent cut-off frequency of the used wavelength, i.e. $k_{c}=\frac{NA_{o}}{\lambda}$. Using a coherent imaging model on the sample’s complex field $s(x)$, which is illuminated with plane waves featuring wave vector $k_{0}$ and imaged by an objective characterized by the coherent point spread function $h_{c}(x)$, the coherent image formation equation reads

[TABLE]

In this equation, $\mathrm{i}\mkern 1.0mu$ is the imaginary unit, $\otimes$ is the convolution operator, and $h_{c}(x)$ is defined as a circle centred on the spatial frequency coordinate origin and with value one inside and zero outside its radius $k_{c}$. The goal of the phase retrieval algorithm is to find the amplitude $a(x)$ and phase $\phi(x)$ component of the complex sample $s(x)=a(x)\times\text{exp}(\mathrm{i}\mkern 1.0mu\phi(x))$. Three pre-processing steps are performed: (1) the raw data is background corrected, (2) then low-pass filtered, and (3) finally an initial guess of the amplitudes $a_{k_{0},k_{c}}(x)$ is made. In analogy to the approach of Zheng[10] this is done by subtraction of a background estimate value $b$, and multiplication of a low-pass filter defined by the support of the incoherent optical transfer function to the image spectra $I_{k_{0},k_{c}}(k)$ to remove noise from outside the pass-band of the objective, i.e. beyond the incoherent cut-off spatial frequency. As the incoherent cut-off frequency is twice the coherent cut-off frequency, a scaled version of the coherent transfer function can be used, i.e. $H_{c}\left(\frac{k}{2}\right)$. Note that Fourier analogues of real space functions, obtained via Fourier transform $\mathfrak{F}$ will be denoted via capitalization, so e.g. $H_{c}(k)=\mathfrak{F}\{h_{c}(x)\}$, with $k$ being the spatial frequency coordinate. The inverse Fourier transform is written as $\mathfrak{F}^{-}$.

After low-pass filtering, the real part $\Re$ of the square root is taken to approximate the field distribution that formed the recorded intensities:

[TABLE]

The phase retrieval part of the algorithm is then initialized using the estimated amplitude of a brightfield image as starting guess $f^{0}(x)$ [13] for the high-resolution Fourier ptychography image. In each iteration up to a total of $n$ rounds, $f^{j}(x)$ is sequentially updated for all available coherent illumination wave vectors $k_{0}$. The sequence is chosen such that the respective sub-sampled parts of Fourier space (which are centred around $k_{0}$ and with radius $k_{c}$), are spiralling out from lower to higher spatial frequencies. Formally in the algorithm, the individual updates are performed in three steps. First, a temporary low-resolution image $t^{j}(x)$ is calculated from the Fourier ptychography estimate $f^{j}(x)$ for the current respective illumination featuring wave vector $k_{0}$, cut-off frequency $k_{c}$ and amplitude transfer function $H_{c}$ as

[TABLE]

The phase $\Phi(t(x))$ of the temporary low-resolution image $t^{j}(x)$ is taken as an estimate of the phase distribution $\phi(x)$ of the sample $s(x)$. Hence, only the amplitude of $t^{j}(x)$ is updated, i.e. replaced by the estimated amplitude $a_{k_{0},k_{c}}(x)$ of the respective pre-processed raw image

[TABLE]

The updated temporary image’s spectrum $T^{j+1}_{k_{0},k_{c}}(k)$ is successively used to replace the respective region in Fourier space of the Fourier ptychography image’s spectrum $F^{j+1}(k)$. This region is centred on $k_{0}$ within a support area defined by the coherent transfer function $H_{c}(k)$ of that respective wavelength:

[TABLE]

After each loop the lower spatial frequencies can be updated using an incoherent brightfield image in analogy to the updating step with evanescent illumination. After $n$ full loops, the final Fourier ptychography image $f(x)$ is produced via apodization and a successive inverse Fourier transform with enlarged Fourier support (potentially made to fit via additional zero-padding) to yield a smoother transform result

[TABLE]

For comparison to brightfield data, an intensity image can be created via squaring of the amplitude part of the Fourier ptychography reconstruction. The presented algorithm was tested on simulated data, as displayed in the top of Figure 2 for a ground truth (GT) input and a successful FP intensity reconstruction.

4 Experimental Results

To validate our approach experimentally, we imaged clusters of metal-organic frameworks (MOFs). The imaged MOFs belong to the group of Zirconium-MOFs with gold nano-rod core and have a small size distribution centered around 200 nm[14]. To ensure adherence of the MOFs to the waveguides, the waveguide chip was plasma-treated for 40 s at 40 W using a 0.35 mbar oxygen atmosphere. Then, a highly diluted aqueous solution of MOFs was drop-casted onto the waveguide chip imaging area and left to dry under a slight angle to provide a more even distribution of the particles. After imaging with the super-condenser, a ground truth image of the sample was generated via atomic force microscopy (AFM) using a commercial system (Bioscope RESOLVE, Bruker). The AFM was operated in tapping mode and RTESPA probes (Bruker) with a nominal spring constant of 6 N/m and resonant frequency 150 kHz. A line scanning resolution of 256 lines with 256 samples/line for 50$\times$50 µm2 was used to generate an overview and 1168 lines with 1168 samples/line for 20$\times$20 µm2 were used for greater detail of selected areas. To counter drift between frames, the individual raw frames were further aligned to each other using semi-automated alignment via the image processing software line ROI image alignment in Fiji[15]. As shown in Figure 3, we retrieved images displaying both enhanced contrast and features beyond the incoherent Abbe diffraction limit.

In panel (a), an overview of the waveguide chip geometry is shown using a brightfield reflectance image (LED illumination centred at 490 nm). As shown in (b), the sample is only barely visible in a 490 nm brightfield incoherent illumination image (for the same objective), which displays an unresolved cluster of 370 nm size in terms of full width at half maximum of a Gaussian fit (data not shown). Displayed in panel (d), the underlying distribution of individual MOFs in this cluster is shown using atomic force microscopy (AFM) and finest scanning reveals dense clustering of particles. The same distribution of clusters that is visible in the AFM image is also present in the Fourier ptychography image in panel (c), which potentially even resolves individual gold nanorod cores in the clusters.

5 Discussion and Conclusion

Care must be taken in interpreting these results as the discernible spots produced by the FP phase retrieval algorithm cannot be associated to individual particles in the AFM recordings with absolute certainty. For instance, although the clusters visible in FP can be mapped onto the AFM ground truth, the individual particles are slightly misplaced or even missing completely. This can have multiple reasons. Firstly, the displacement might be traced back to drift of the sample during raw data acquisition. A shift of sample information even in the nanometer range might thus produce artefacts causing erroneous particle localization. Although drift was compensated for computationally, it cannot be avoided completely. Secondly, the gold core of the MOFs is asymmetric and cause scattering preferentially in certain directions. Paired with variable signal strength for different illuminations caused by higher losses of the waveguides in longer arms, this could lead to some particles outshining others in the reconstruction. Also note that the amplitude of the scattered light is wavelength dependent. Despite trying to adjust for this via variation of the laser illumination intensity and the exposure time of the camera, a limited maximum number of photons in certain raw frames restricted the achievable signal-to-noise ratio (SNR). Residual wavelength-dependent waveguide autofluorescence increases the challenge of limited SNR additionally. In the future, more advanced algorithms originally developed for conventional Fourier ptychography could be adopted to computationally alleviate some of these concerns[16]. Furthermore, axial chromatic offset could not be accounted for, which might stem from residual imperfections of the employed apochromatic objective and achromatic tube lens. This might be tackled via fine z-stepping of the objective (not possible in the presented set-up which offers no finer than 0.5 µm steps) and post-acquisition alignment or pre-acquisition calibration (potentially using multiple cameras to increase acquisition speeds). Still, the produced images are promising for the young field of labelfree chip-nanoscopy.

Looking ahead, a multitude of further developments might appear in close reach, which are in parts also conceptually simple. An example would be a change of the chip material. The catch here is the unavailability of established production methods for single-mode waveguides of more exotic character. Nevertheless, promising candidates for alternative waveguide materials that also keep propagation losses low at short wavelengths are Ta2O5 [17], TiO2 [18], or graphene [19]) - see Figure 4. TiO2 is an especially interesting material, as it transmits even in parts of the near ultra-violet range, which would allow sub-100nm resolution:

[TABLE]

A further current bottleneck, the limited field of view, could be alleviated through use of a slab region at the imaging area illuminated by un-tapered single-mode waveguides. Although initially propagating as circular waves (the two-dimensional analog of spherical waves produced by free-space point sources), the waves emanating from the waveguide outlets would be sufficiently close to plane waves already after a travelled distance of few wavelengths and thus suitable for quasi-coherent imaging. For a further field of view enhancement, illumination from a limited number of directions (potentially even only single-sided) could be feasible for amplitude-only samples[20] and would simplify both waveguide geometry and deliverable powers to the imaging area due to reduced bending losses. More speculatively, on-chip lasers[21] are an option to simplify the microscopy set-up and circumvent coupling losses. An intriguing alternative could further be the use of a broadly emitting fluorescent film to generate the illumination light with successive narrow-band emission filtering as proposed by Pang et al[22]. Although the illumination in this set-up is not coherent, FP algorithms have been developed that can be applied[13]. To tackle drift of the sample during image acquisition, a fully automated coupling procedure might speed up slow manual coupling of various wavelengths and to different inputs. A possible future improvement in this respect is input-multiplexing which is feasible via conventional fiber-array adaptors. These are standard in the telecommunication industry but are to date mostly available for infrared wavelengths.

In conclusion, we have developed a labelfree nanoscopy method that combines Fourier ptychography with waveguide microscopy to realize a super-condenser. The waveguide geometry allows the use of maximally inclined coherent darkfield illumination and additionally makes use of the large refractive index of Si3N4 as waveguide material to further double the illumination wave vector amplitudes as compared to air. We demonstrated our method in silico and validated it in experimental imaging of metal organic frameworks that contained gold nano-rod cores. As proven by atomic force microscopy, we were able to image MOF clusters successfully and could even infer the distribution of individual particles within the clusters. Taken together, Fourier ptychography in combination with enlarged illumination wave vectors is a promising avenue to enable nanoscopic imaging without the requirement of extrinsic labels.

Funding

FS acknowledges funding from the European Molecular Biology Organisation (#7411) and Marie Skłodowska-Curie actions (#836355). BSA acknowledges funding from the European Research Council (#336716). CFK acknowledges funding from the Physical Sciences Research Council (EP/H018301/1), Medical Research Council (MR/K015850/1 and MR/K02292X/1), Wellcome Trust (089703/Z/09/Z), and Infinitus China Ltd.

Author contributions statement

FS and BSA conceived the idea of waveguide-chip based label-free nanoscopy using Fourier ptychography. FS further expanded the idea towards the use of different wavelengths to fill Fourier space, performed simulations and analyzed the data. JCT and FTD designed, produced, and characterized the waveguides. ISO prepared samples. FS and ISO built the chip-microscope and performed waveguide imaging. IM performed AFM measurements. JWMO provided metal-organic frameworks. FS wrote the first version of the manuscript and all authors contributed towards the writing of the manuscript. CFK and BSA provided research tools and supported the project.

Disclosures

BSA has applied for patent GB1705660.7 on an Optical component for generating a periodic light pattern that covers the designs of the waveguide chip used in this work. The other authors declare no competing interests.