Spatio-angular fluorescence microscopy I. Basic theory

TL;DR

This paper develops a foundational spatio-angular theoretical framework for fluorescence microscopy, modeling imaging systems as linear operators and introducing the dipole spatio-angular transfer function to unify analysis of single and ensemble dipole imaging.

Contribution

It introduces a novel spatio-angular theory and a dipole transfer function, providing a unified mathematical framework for fluorescence microscopy analysis.

Findings

Modeling fluorescence microscopy as a linear Hilbert-space operator.

Introduction of the dipole spatio-angular transfer function.

Framework applicable to both single and ensemble dipole imaging.

Abstract

We introduce the basic elements of a spatio-angular theory of fluorescence microscopy, providing a unified framework for analyzing systems that image single fluorescent dipoles and ensembles of overlapping dipoles that label biological molecules. We model an aplanatic microscope imaging an ensemble of in-focus fluorescent dipoles as a linear Hilbert-space operator, and we show that the operator takes a particularly convenient form when expressed in a basis of complex exponentials and spherical harmonics$-$a form we call the dipole spatio-angular transfer function. We discuss the implications of our analysis for all quantitative fluorescence microscopy studies and lay out a path towards a complete theory.

Equations

$$\mathbf{g}=\mathcal{H}\mathbf{f}.$$

$$g^{\prime }({\mathbf{r}}_d^{\prime })={\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_o{h}^{\prime }({\mathbf{r}}_d^{\prime },{\mathbf{r}}_o)f({\mathbf{r}}_o),$$

$$g^{\prime }({\mathbf{r}}_d^{\prime })={\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_o{h}^{\prime }({\mathbf{r}}_d^{\prime }-m{\mathbf{r}}_o)f({\mathbf{r}}_o),$$

$$h({\mathbf{r}}_d-{\mathbf{r}}_o)=\frac{h^{\prime }(m[{\mathbf{r}}_d-{\mathbf{r}}_o])}{P_{\text{mono}}},$$

$$g({\mathbf{r}}_d)=\frac{g^{\prime }(m{\mathbf{r}}_d)}{P_{\text{mono}}}.$$

$$g({\mathbf{r}}_d)={\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_{o}h({\mathbf{r}}_d-{\mathbf{r}}_o)f({\mathbf{r}}_o).$$

$${\int }_{{\mathbb{R}}^2}d\mathbf{r}h(\mathbf{r})=1.$$

$$G(\mathbf{\nu })=H(\mathbf{\nu })F(\mathbf{\nu }),$$

$$G(\mathbf{\nu })={\int }_{{\mathbb{R}}^2}d\mathbf{r}g(\mathbf{r})\text{exp}(-2\pi i\mathbf{r}\cdot \mathbf{\nu }),$$

$$H(\mathbf{\nu })={\int }_{{\mathbb{R}}^2}d\mathbf{r}h(\mathbf{r})\text{exp}(-2\pi i\mathbf{r}\cdot \mathbf{\nu }),$$

$$F(\mathbf{\nu })={\int }_{{\mathbb{R}}^2}d\mathbf{r}f(\mathbf{r})\text{exp}(-2\pi i\mathbf{r}\cdot \mathbf{\nu }).$$

$$|c({\mathbf{r}}_d-{\mathbf{r}}_o){|}^2=h({\mathbf{r}}_d-{\mathbf{r}}_o).$$

$$H(\mathbf{\nu })={\int }_{{\mathbb{R}}^2}d\mathbf{\tau }C(\mathbf{\tau })C^{\ast }(\mathbf{\tau }-\mathbf{\nu }),$$

$$C(\mathbf{\tau })={\int }_{{\mathbb{R}}^2}d\mathbf{r}c(\mathbf{r})\text{exp}[-2\pi i\mathbf{r}\cdot \mathbf{\tau }].$$

$$g^{\prime }({\mathbf{r}}_d^{\prime })={\int }_{{\mathbb{S}}^2}d\text{textso}{\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_o{h}^{\prime }({\mathbf{r}}_d^{\prime },{\mathbf{r}}_o,\text{textso})f({\mathbf{r}}_o,\text{textso}),$$

$$g^{\prime }({\mathbf{r}}_d^{\prime })={\int }_{{\mathbb{S}}^2}d\text{textso}{\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_o{h}^{\prime }({\mathbf{r}}_d^{\prime }-m{\mathbf{r}}_o,\text{textso})f({\mathbf{r}}_o,\text{textso}).$$

$$h({\mathbf{r}}_d-{\mathbf{r}}_o,\text{textso})=\frac{h^{\prime }(m[{\mathbf{r}}_d-{\mathbf{r}}_o],\text{textso})}{P_{\text{dip}}},$$

$$g({\mathbf{r}}_d)=\frac{g^{\prime }(m{\mathbf{r}}_d)}{P_{\text{dip}}}.$$

$$g({\mathbf{r}}_d)={\int }_{{\mathbb{S}}^2}d\text{textso}{\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_{o}h({\mathbf{r}}_d-{\mathbf{r}}_o,\text{textso})f({\mathbf{r}}_o,\text{textso}).$$

$${\int }_{{\mathbb{S}}^2}d\text{textso}{\int }_{{\mathbb{R}}^2}d\mathbf{r}h(\mathbf{r},\text{textso})=1.$$

$$G(\mathbf{\nu })={\int }_{{\mathbb{S}}^2}d\text{textso}H(\mathbf{\nu },\text{textso})F(\mathbf{\nu },\text{textso}),$$

$$H(\mathbf{\nu },\text{textso})$$

$$F(\mathbf{\nu },\text{textso})$$

$${\int }_{{\mathbb{S}}^2}d\hat{\mathbf{s}}p(\hat{\mathbf{s}})q^{\ast }(\hat{\mathbf{s}})=\sum _{\ell =0}^{\infty }\sum _{m=-\ell }^{\ell }{P}_{\ell }^m{Q}_{\ell }^{m\ast },$$

$$P_{\ell }^m={\int }_{{\mathbb{S}}^2}d\hat{\mathbf{s}}p(\hat{\mathbf{s}})Y_{\ell }^{m\ast }(\hat{\mathbf{s}}),$$

$$g({\mathbf{r}}_d)=\sum _{\ell =0}^{\infty }\sum _{m=-\ell }^{\ell }{\int }_{{\mathbb{R}}^2}d{\mathbf{r}}_o{H}_{\ell }^m({\mathbf{r}}_d-{\mathbf{r}}_o)F_{\ell }^m({\mathbf{r}}_o),$$

$$H_{\ell }^m({\mathbf{r}}_d-{\mathbf{r}}_o)={\int }_{{\mathbb{S}}^2}d\text{textso}h({\mathbf{r}}_d-{\mathbf{r}}_o,\text{textso})Y_{\ell }^{m\ast }(\text{textso}),$$

$$F_{\ell }^m({\mathbf{r}}_o)={\int }_{{\mathbb{S}}^2}d\text{textso}f({\mathbf{r}}_o,\text{textso})Y_{\ell }^{m\ast }(\text{textso}).$$

$$G(\mathbf{\nu })=\sum _{\ell =0}^{\infty }\sum _{m=-\ell }^{\ell }{\mathsf{H}}_{\ell }^m(\mathbf{\nu }){\mathsf{F}}_{\ell }^m(\mathbf{\nu }),$$

$${\mathsf{H}}_{\ell }^m(\mathbf{\nu })$$

Full text

Spatio-angular fluorescence microscopy

I. Basic theory

Talon Chandler

  

\authormark1,* Hari Shroff

  

\authormark2,3 Rudolf Oldenbourg

  

\authormark3 and Patrick La Rivière\authormark1,3

\authormark1University of Chicago, Department of Radiology, Chicago, Illinois 60637, USA

\authormark2Section on High Resolution Optical Imaging, National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Bethesda, Maryland 20892, USA

\authormark3Marine Biological Laboratory, Bell Center, Woods Hole, Massachusetts 02543, USA

\authormark*talonchandler@talonchandler.com

††journal: osajournal††articletype: Research Article

{abstract*}

We introduce the basic elements of a spatio-angular theory of fluorescence microscopy, providing a unified framework for analyzing systems that image single fluorescent dipoles and ensembles of overlapping dipoles that label biological molecules. We model an aplanatic microscope imaging an ensemble of in-focus fluorescent dipoles as a linear Hilbert-space operator, and we show that the operator takes a particularly convenient form when expressed in a basis of complex exponentials and spherical harmonics—a form we call the dipole spatio-angular transfer function. We discuss the implications of our analysis for all quantitative fluorescence microscopy studies and lay out a path towards a complete theory.

1 Introduction

Fluorescence microscopes are widely used in the biological sciences for imaging fluorescent molecules that label specific proteins and biologically important molecules. While most fluorescence microscopy experiments are designed to measure only the spatial distribution of fluorophores, a growing number of experiments seek to measure both the spatial and angular distributions of fluorophores by the use of polarizers [1, 2, 3, 4, 5] or point spread function engineering [6, 7].

Meanwhile, single-molecule localization microscopy (SMLM) experiments use spatially sparse fluorescent samples to localize single molecules with precision that surpasses the diffraction limit. Noise limits the precision of this localization [8, 9], and several studies have shown that model mismatch (e.g. ignoring the effects of vector optics, dipole orientation [10], and dipole rotation [11]) can introduce localization bias as well. Therefore, the most precise and accurate SMLM experiments must use an appropriate model and jointly estimate both the position and orientation of each fluorophore. Several studies have successfully used vector optics and dipole models to estimate the position and orientation of single molecules [12, 13, 14, 15, 16], and there is growing interest in designing optical systems for measuring the position, orientation, and rotational dynamics of single molecules [6, 17, 18, 7, 19].

While many studies have focused on improving imaging models for spatially sparse fluorescent samples, we consider the more general case and aim to improve imaging models for arbitrary samples including those containing ensembles of fluorophores within a resolvable volume. In particular, we examine the effects of two widely used approximations in fluorescence microscopy—the monopole approximation and the scalar approximation.

We use the term monopole to refer to a model of a fluorophore that treats it as an isotropic absorber/emitter. Although the term monopole approximation is not in widespread use, we think it accurately describes the way many models of fluorescence microscopy treat fluorophores, and we use the term to distinguish the monopole model from more realistic dipole and higher-order models. Despite their use in models, electromagnetic monopole absorber/emitters do not exist in nature. All physical fluorophores absorb and emit radiation with dipole or higher-order moments, and these moments are always oriented in space. For a classical mental model of fluorophores we imagine each dipole as a small oriented antenna (with incoherent absorption and emission moments) where electrons are constrained to move along a single direction.

All fluorescence microscopy models that use an optical point spread function or an optical transfer function to describe the mapping between the fluorophores and the irradiance on the detector implicitly make the monopole approximation. The optical point spread function is the irradiance response of an optical system to an isotropic point source, so it cannot model the response due to an anisotropic dipole radiator. In this work we define monopole and dipole transfer functions that describe the mapping between fluorophores and the measured irradiance. Although optical systems are an essential part of microscopes, fluorescence microscopists are interested in measuring the properties of fluorophores (not optics), so the monopole and dipole transfer functions are more directly useful than the optical transfer function for the problems that fluorescence microscopists are interested in solving.

While the monopole approximation applies to the fluorescent object, the scalar approximation applies to the fields that propagate through the microscope. Modeling the electric fields in a region requires a three-dimensional vector field, but if the electric fields are random or completely parallel, a scalar field is sufficient, and we can replace the vector-valued electric field, $\mathbf{E}$, with a scalar-valued field, $U$.

The scalar approximation is often made together with the monopole approximation. For example, the Born-Wolf model [20] and the Gibson-Lanni model [21] make both the monopole and scalar approximations when applied to fluorescence microscopes. However, some models make the monopole approximation but not the scalar approximation. For example, the Richards-Wolf model [22] considers the role of vector-valued fields in the optical system, but it is an optical model so when it is applied to fluorescence microscopes the monopole approximation is assumed.

This work lies at the intersection of three subfields of fluorescence microscopy: (1) spatial ensemble imaging where each resolvable volume contains many fluorophores and the goal is to find the concentration of fluorophores as a function of position in the sample, (2) spatio-angular ensemble imaging where each resolvable volume contains many fluorophores and the goal is to find the concentration and average orientation of fluorophores as a function of position in the sample, and (3) SMLM imaging where fluorophores are sparse in the sample and the goal is to find the position and orientation of each fluorophore. We briefly review how these three subfields use the monopole and scalar approximations.

The large majority of fluorescence microscopes are used to image ensembles of fluorophores, and most existing modeling techniques make use of the monopole approximation, the scalar approximation, or both. As discussed above, the Gibson-Lanni model, the Born-Wolf model, and the Richards-Wolf model are approximate when applied to fluorescence microscopy data because they only model monopole emitters. Deconvolution algorithms that use these models may make biased estimates of fluorophore concentrations since they ignore the dipole excitation and emission of fluorophores.

A small but growing group of microscopists is interested in measuring the orientation and position of ensembles of fluorophores [1, 2, 3, 4, 5]. These techniques typically use polarizers to make multiple measurements of the same object with different polarizer orientations, then they use a model of the dipole excitation and emission processes [23] to recover the orientation of fluorophores using pixel-wise arithmetic. Although these studies do not adopt the scalar or monopole approximations for the angular part of the problem, they adopt both approximations when they consider the spatial part of the problem. Existing works either ignore the spatial reconstruction problem [1, 2, 3, 4] or assume that the spatial and angular reconstruction problems can be solved sequentially [5].

The most precise experiments in SMLM imaging do not adopt the scalar or monopole approximations. Although many works have applied dipole models with vector optics, fewer have considered the effects of rotational or spatial diffusion, and to our knowledge no works have considered both rotational and spatial diffusion together. We will see that the dipole transfer functions are useful tools for incorporating angular and spatial diffusion into SMLM simulations and reconstructions.

In the present work we begin to place these three subfields on a common theoretical footing. First, in section 2, we consider arbitrary fluorescence imaging models and lay out a plan for developing a model for spatio-angular imaging. In section 3 we review the familiar monopole imaging model, and in section 4 we extend the model to dipoles. Finally, in section 5 we discuss the results and their broader implications.

In this paper we focus on modeling a single-view fluorescence microscope without polarizers. In future papers of this series we will extend our models to include polarizers and multi-view microscopes. Additionally, we have restricted this paper to the forward problem—the mapping between a known object and the data. In future papers we will consider the inverse problem, and the singular value decomposition (SVD) will play a central role.

2 Theory

We begin our analysis with the abstract Hilbert space formalism of Barrett and Myers [24, ch. 1.3]. Our first task is to formulate the imaging process as a mapping between two Hilbert spaces $\mathcal{H}:\mathbb{U}\rightarrow\mathbb{V}$, where $\mathbb{U}$ is a set that contains all possible objects, $\mathbb{V}$ is a set that contains all (possibly noise-corrupted) datasets, and $\mathcal{H}$ is a model of the instrument that maps between these two spaces. We denote (possibly infinite-dimensional) Hilbert-space vectors in $\mathbb{U}$ with $\mathbf{f}$, Hilbert-space vectors in $\mathbb{V}$ with $\mathbf{g}$, and the mapping between the spaces with

[TABLE]

Throughout this work we will use the letters $g$, $h$, and $f$ with varying fonts, capitalizations, and arguments to represent the data, the instrument, and the object, respectively.

Once we have identified the spaces $\mathbb{U}$ and $\mathbb{V}$, we can start expressing the mapping between the spaces in a specific object-space and data-space basis. In most cases the easiest mapping to find uses a delta-function basis—we expand object and data space into delta functions, then express the mapping as an integral transform. After finding this mapping we can start to investigate the same mapping in different bases.

The above discussion is quite abstract, but it is a powerful point of view that will enable us to unify the analysis of spatio-angular fluorescence imaging. In section 3 we will demonstrate the formalism by examining a familiar monopole imaging model, and we will demonstrate the mapping between object and data space in two different bases. In section 4 we will extend the monopole imaging model to dipoles and examine the mapping in four different bases.

3 Monopole imaging

We start by considering a microscope that images a field of in-focus monopoles by recording the irradiance on a two-dimensional detector. This section treads familiar ground, but it serves to establish the concepts and notation that will be necessary when we extend to the dipole case.

We can represent the object as a function that assigns a real number to each point on a plane, so we identify object space as $\mathbb{U}=\mathbb{L}_{2}(\mathbb{R}^{2})$—the set of square-integrable functions on the two-dimensional plane. Similarly, we have a two-dimensional detector that measures a real number at each point on a plane, so data space is the same set $\mathbb{V}=\mathbb{L}_{2}(\mathbb{R}^{2})$.

Next, we name the representations of our object and data in a specific basis. In a delta function basis the object can be represented by a function $f(\mathbf{\mathbf{r}}_{o})$ called the monopole density—the number of monopoles per unit area at the two-dimensional position $\mathbf{\mathbf{r}}_{o}$. Similarly, in a delta function basis the data can be represented by a function $g^{\prime}(\mathbf{r}_{d}^{\prime})$ called the irradiance—the power received by a surface per unit area at position $\mathbf{r}_{d}^{\prime}$. Note that we have adopted a slightly unusual convention of using primes to denote unscaled coordinates. Later in this section we will introduce unprimed scaled coordinates that we will use throughout the rest of the paper.

A reasonable starting point is to assume that the relationship between the object and the data is linear—this is true in many fluorescence microscopes because fluorophores emit incoherently, so a scaled sum of fluorophores will result in a scaled sum of the irradiance patterns created by the individual fluorophores. Note that our assumption of linearity excludes cases where fluorophores interact (e.g. homoFRET) or saturate (e.g. non-linear fluorescence microscopy).

If the mapping is linear, we can write the irradiance as a weighted integral over a field of monopoles

[TABLE]

where $h^{\prime}(\mathbf{r}_{d}^{\prime},\mathbf{\mathbf{r}}_{o}{})$ is the irradiance at position $\mathbf{r}_{d}^{\prime}$ created by a point source at $\mathbf{\mathbf{r}}_{o}$.

Next, we assume that the optical system is aplanatic—Abbe’s sine condition is satisfied and on-axis points are imaged without aberration. Abbe’s sine condition guarantees that off-axis points are imaged without spherical aberration or coma [25, ch. 1], so the imaging system can be modeled within the field of view of the optical system as a magnifier with shift-invariant blur

[TABLE]

where $m$ is a magnification factor.

We can simplify our analysis by changing coordinates and writing Eq. (3) as a convolution [24, ch. 7.2.7]. We define a demagnified detector coordinate $\mathbf{r}_{d}=\mathbf{r}_{d}^{\prime}/m$ and a normalization factor that corresponds to the total power incident on the detector plane due to a point source $P_{\text{mono}}=\int_{\mathbb{R}^{2}}d\mathbf{r}\,h^{\prime}(m\mathbf{r})$ where $\mathbf{r}=\mathbf{r}_{d}-\mathbf{\mathbf{r}}_{o}$. We use these scaling factors to define the monopole point spread function as

[TABLE]

and the scaled irradiance as

[TABLE]

With these definitions we can express the mapping between the object and the data as a familiar convolution

[TABLE]

We have chosen to normalize the monopole point spread function so that

[TABLE]

The monopole point spread function corresponds to a measurable irradiance, so it is always real and positive.

The mapping between the object and the data in a linear shift-invariant imaging system takes a particularly simple form in a complex exponential (i.e. Fourier) basis. If we apply the Fourier convolution theorem to Eq. (6) we find that

[TABLE]

where we define the scaled irradiance spectrum as

[TABLE]

the monopole transfer function as

[TABLE]

and the monopole spectrum as

[TABLE]

The monopole point spread function is normalized and real, so we know that the monopole transfer function is normalized, $H(0)=1$, and conjugate symmetric, $H(-\bm{\nu})=H^{*}(\bm{\nu})$, where $z^{*}$ denotes the complex conjugate of $z$.

Notice that Eqs. (6) and (8) are expressions of the same mapping between object and data space in different bases. Figure 1 summarizes the relationship between object and data space in both bases.

We have been careful to use the term monopole transfer function instead of the commonly-used term optical transfer function. We reserve the term optical transfer function for optical systems—the optical transfer function maps between an input irradiance spectrum and an output irradiance spectrum in an optical system. We can use optical transfer functions to model the propagation of light through a microscope, but ultimately we are always interested in the object, not the light emitted by the object. We will find the distinction between the optical transfer function and the object transfer function to be especially valuable when we consider dipoles in section 4.

3.1 Monopole coherent transfer functions

Although the Fourier transform can be used to calculate the monopole transfer function directly from the monopole point spread function, there is a well-known alternative that exploits coherent transfer functions. The key idea is that the monopole point spread function can always be written as the absolute square of a scalar-valued monopole coherent spread function, $c(\mathbf{r}_{d}-\mathbf{\mathbf{r}}_{o})$, defined by

[TABLE]

Physically, the monopole coherent spread function corresponds to the scalar-valued field on the detector with appropriate scaling.

We can plug Eq. (12) into Eq. (10) and use the autocorrelation theorem to rewrite the monopole transfer function as

[TABLE]

where we have introduced the monopole coherent transfer function as the two-dimensional Fourier transform of the monopole coherent spread function:

[TABLE]

Physically, the monopole coherent transfer function corresponds to the scalar-valued field in a Fourier plane of the detector with appropriate scaling.

The coherent transfer function provides a valuable shortcut for analyzing microscopes since it is often straightforward to calculate the field in a Fourier plane of the detector. A typical approach for calculating the transfer functions is to (1) calculate the field in a Fourier plane of the detector, (2) scale the field to find the monopole coherent transfer function, then (3) use the relationships in Fig. 2 to calculate the other transfer functions.

4 Dipole imaging

Now we consider a microscope imaging a field of in-focus dipoles by recording the irradiance on a two-dimensional detector. A function that assigns a real number to each point on a plane is not sufficient to specify a field of dipoles because the dipoles can have different orientations. To represent the object we need to extend object space to $\mathbb{U}=\mathbb{L}_{2}(\mathbb{R}^{2}\times\mathbb{S}^{2})$—the set of square-integrable functions on the product space of a plane and a two-dimensional sphere (the usual sphere embedded in $\mathbb{R}^{3}$). To visualize functions in object space we imagine a sphere at every point on a plane with a scalar value assigned to every point on each sphere.

In a delta function basis the object can be represented by a function $f(\mathbf{\mathbf{r}}_{o},\textso{)}$ called the dipole density—the number of dipoles per unit area and per unit solid angle at position $\mathbf{\mathbf{r}}_{o}{}$ and oriented along . Similar to the monopole case, we model the mapping between the object and the irradiance in a delta function basis as an integral transform

[TABLE]

where $h^{\prime}(\mathbf{r}_{d}^{\prime},\mathbf{\mathbf{r}}_{o},\textso{)}$ is the irradiance at position $\mathbf{r}_{d}^{\prime}$ created by a point source at $\mathbf{\mathbf{r}}_{o}$ with orientation \textso{}.Noticethatwehaveconsideredallpossibleorientations$and integrated over the sphere$\mathbb{S}^{2}$. The dipole density is always symmetric under angular inversion,$f(\mathbf{\mathbf{r}}{o},\textso{)}=f(\mathbf{\mathbf{r}}{o},-\textso{)}$, so we could have chosen to integrate over a hemisphere and adjusted the definition of the dipole density by a factor of two. For convenience we will continue to integrate over the complete sphere. We note that all functions in this work with$\textso{}asanindependentvariablearesymmetricunderangularinversion,→-.

If the optical system is aplanatic, we can write the integral transform as

[TABLE]

We define the same demagnified detector coordinate $\mathbf{r}_{d}=\mathbf{r}_{d}^{\prime}/m$ and a new normalization factor that corresponds to the total power incident on the detector due to a spatial point source with an angularly uniform distribution of dipoles $P_{\text{dip}}=\int_{\mathbb{S}^{2}}d\textso{\int}_{\mathbb{R}^{2}}d\mathbf{r}\,h^{\prime}(m\mathbf{r},\textso{)}$. We use these scaling factors to define the dipole point spread function as

[TABLE]

and the scaled irradiance as

[TABLE]

With these definitions we can express the mapping between the object and the data as

[TABLE]

Equation (19) is a key result because it represents the mapping between object space and data space in a delta function basis. The integrals in Eq. (19) would be extremely expensive to compute for an arbitrary object, but the integrals simplify to an efficient sum if the object is spatially and angularly sparse. In other words, Eq. (19) is ideal for simulating and analyzing single fluorophores that are rigidly attached to an oriented structure.

Similar to the monopole case, we have chosen to normalize the dipole point spread function so that

[TABLE]

The dipole point spread function is a measurable quantity, so it is real and positive.

4.1 Dipole spatial transfer function

We can make our first change of basis by applying the Fourier-convolution theorem to Eq. (19), which yields

[TABLE]

where we define the dipole spatial transfer function as

[TABLE]

and the dipole spatial spectrum as

[TABLE]

Since the dipole point spread function is normalized and real, we know that the dipole spatial transfer function is normalized, $\int_{\mathbb{S}^{2}}d\mathbf{\hat{s}}\,H(0,\textso{)}=1$, and conjugate symmetric, $H(-\bm{\nu},\textso{)}=H^{*}(\bm{\nu},\textso{)}$.

This basis is ideal for simulating and analyzing objects that are angularly sparse and spatially dense; e.g. rod-like structures that contain fluorophores in a fixed orientation, or rotationally fixed fluorophores that are undergoing spatial diffusion.

4.2 Dipole angular transfer function

The spherical harmonics are another set of convenient basis functions that play the same role as complex exponentials in spatial transfer functions—see Appendix A for an introduction to the spherical harmonics. We can change basis from spherical delta functions to spherical harmonics by applying the generalized Plancherel theorem for spherical functions

[TABLE]

where $p(\mathbf{\hat{s}})$ and $q(\mathbf{\hat{s}})$ are arbitrary functions on the sphere, $P_{\ell}^{m}$ and $Q_{\ell}^{m}$ are their spherical Fourier transforms defined by

[TABLE]

and $Y_{\ell}^{m}(\mathbf{\hat{s}})$ are the spherical harmonic functions defined in Appendix A. Equation (24) expresses the fact that scalar products are invariant under a change of basis [24, Eq. 3.78]. The left-hand side of Eq. (24) is the scalar product of $\mathbb{L}_{2}(\mathbb{S}^{2})$ functions in a delta function basis and the right-hand side is the scalar product of $\mathbb{L}_{2}(\mathbb{S}^{2})$ functions in a spherical harmonic function basis. Applying Eq. (24) to Eq. (19) yields

[TABLE]

where we have defined the dipole angular transfer function as

[TABLE]

and the dipole angular spectrum as

[TABLE]

Since the dipole point spread function is normalized and real, we know that the dipole angular transfer function is normalized, $\int_{\mathbb{R}^{2}}d\mathbf{r}\,H_{0}^{0}(\mathbf{r})=1$, and conjugate symmetric, $H_{\ell}^{-m}(\mathbf{r})=(-1)^{m}H_{\ell}^{m*}(\mathbf{r})$.

This basis is well suited for simulating and analyzing objects that are spatially sparse and angularly dense; e.g. single fluorophores that are undergoing angular diffusion, or many fluorophores that are within a resolvable volume with varying orientations.

4.3 Spatio-angular dipole transfer function

We can arrive at our final basis in two ways: by applying the generalized Plancherel theorem for spherical functions to Eq. (21) or by applying the Fourier convolution theorem to Eq. (26). We follow the first path and find that

[TABLE]

where we have defined the dipole spatio-angular transfer function as

[TABLE]

and the dipole spatio-angular spectrum as

[TABLE]

Since the dipole point spread function is normalized and real, we know that the dipole spatio-angular transfer function is normalized, $\mathsf{H}_{0}^{0}(0)=1$, and conjugate symmetric, $\mathsf{H}_{\ell}^{-m}(-\bm{\nu})=(-1)^{m}\mathsf{H}_{\ell}^{m*}(\bm{\nu})$.

This basis is well suited for simulating and analyzing arbitrary samples because it exploits the band limit of the imaging system. We note that most single molecule imaging experiments are best described in this basis because of the effects of spatial and rotational diffusion.

Figure 3 summarizes the relationships between the four bases that we can use to compute the image of a field of dipoles. We reiterate that all four bases may be useful depending on the sample.

4.4 Dipole coherent transfer functions

Similar to the monopole case, there is an efficient way to calculate the transfer functions using coherent transfer functions. The dipole point spread function can always be written as the absolute square of a vector-valued function, $\mathbf{c}(\mathbf{r}_{d}-\mathbf{\mathbf{r}}_{o},\textso{)}$, called the dipole coherent spread function:

[TABLE]

Physically, the dipole coherent spread function corresponds to the vector-valued electric field on the detector with appropriate scaling. We need a vector-valued coherent transfer function since the polarization of the field plays a significant role in dipole imaging, so the dipole point spread function cannot be written as an absolute square of a scalar-valued function.

We can plug Eq. (32) into Eq. (22) and use the autocorrelation theorem to rewrite the dipole spatial transfer function as

[TABLE]

where we have introduced the dipole coherent transfer function $\mathbf{C}(\bm{\tau},\textso{)}$ as the two-dimensional Fourier transform of the dipole coherent spread function:

[TABLE]

Physically, the dipole coherent transfer function corresponds to the vector-valued electric field created by a dipole oriented along \textso{}inaFourierplaneofthedetectorwithappropriatescaling.Similartothemonopolecase,wecancalculatethedipole-orientation-dependentfieldsinaFourierplaneofthedetector,scaleappropriatelytofindthedipolecoherenttransferfunction,thenusetherelationshipsinFig.\ref{fig:transfer-functions}tocalculatetheothertransferfunctions.Finally,wenotethatthedipolecoherenttransferfunctionisidentical(uptoscalingfactors)towhatAgrawalet.al.callthe\textit{Green's tensor}\cite[cite]{[@@bibref{}{agrawal2012}{}{}]}andNovotnyandHecht^{\prime}s\textit{dyadic point spread function}multipliedbythedipolemomentvector\cite[cite]{[@@bibref{}{nov2006}{}{}]}.\par$

5 Discussion

5.1 When are dipole transfer functions necessary?

Model mismatch can lead to biased estimates of the position and orientation of fluorophores, so the most accurate fluorescence microscopy experiments will always use dipole transfer functions over monopole transfer functions. However, in many practical situations noise will mask the effects of vector optics and dipoles. If the fluorophores are rotationally unconstrained or there are many randomly oriented fluorophores in a resolvable volume—common situations in biological applications—then the effects of vector optics and dipoles will be masked by noise in all but the highest-SNR regimes. Therefore, the dipole transfer functions are most broadly useful when the sample contains fluorophores that are rotationally constrained. As rotational constraints increase, the effects of vector optics and dipoles will become apparent in lower SNR regimes. In the next paper of this series we will calculate the dipole transfer functions for a 4-$f$ imaging system and investigate the conditions under which the monopole and dipole models are identical.

5.2 Alternative transfer functions

Throughout this work we have used the spherical harmonic functions as a basis for functions on the sphere, but there are other basis functions that can be advantageous in some cases. Several works [15, 17, 27, 7, 19] have used the second moments as basis functions for the sphere because they arise naturally when computing the dipole point spread function. These works use an alternative to the dipole angular transfer function that uses the second moments as basis functions so the forward model can be written as

[TABLE]

where

[TABLE]

and $Z_{j}(\mathbf{\hat{s}})=\{s_{x}^{2},s_{y}^{2},s_{z}^{2},s_{x}s_{y},s_{y}s_{z},s_{x}s_{z}\}$ are the second moments. This formulation is similar to the dipole angular transfer function approach because it can exploit the spatial sparsity of the sample, but it does not require a cumbersome expansion of the dipole point spread function onto spherical harmonics.

However, the spherical harmonics provide several advantages over the second moments. First, the spherical harmonics form a complete basis for functions on the sphere, while the second moments span a much smaller function space. The usual approach to extending the span of the second moments is to use the fourth (or higher) moments, but this extension requires a completely new set of basis functions while the spherical harmonics can be extended by simply adding higher order terms. Second, the spherical harmonics are orthonormal, which will allow us to deploy invaluable tools from linear algebra—linear subspaces, rank, SVD, etc.— to analyze and compare microscope designs. Finally, using the spherical harmonics provides access to a set of fast algorithms. The naive expansion of an arbitrary discretized $N$ point spherical function onto spherical harmonics (or second moments) requires a $\mathcal{O}(N^{2})$ matrix multiplication, while pioneering work by Driscoll and Healy [28] showed that the forward discrete fast spherical harmonic transform can be computed with a $\mathcal{O}(N(\log N)^{2})$ algorithm and its inverse can be computed with a $\mathcal{O}(N^{3/2})$ algorithm. To our knowledge no similarly fast algorithms exist for expansion onto the higher-order moments.

Zhenghao et. al. [5] have used the circular harmonics to model the orientation of dipoles. The circular harmonics are complete and orthogonal, but they artificially restrict the reconstructed dipoles to the transverse plane of the microscope—a rare situation in real experiments.

The diffusion magnetic resonance imaging community uses both the second moments (or second-order tensor) basis functions [29] and the spherical harmonic basis functions [30]. Descoteaux et. al. have provided an explicit transformation matrix to convert between these basis functions [31].

5.3 Towards spatio-angular reconstructions

We have focused on modeling the mapping between the object and the data in this paper, but ultimately we are interested in reconstructing the object from the data. Applying the monopole approximation simplifies the reconstruction problem because both object and data space are $\mathbb{L}_{2}(\mathbb{R}^{2})$, so we can directly apply regularized inverse filters and maximum likelihood methods. The dipole model expands object space to $\mathbb{L}_{2}(\mathbb{R}^{2}\times\mathbb{S}^{2})$, so the inverse problem becomes much more challenging. In future work we will use the singular value decomposition to find inverse filters, and we will consider using polarizers and multiple views to increase the size of data space.

6 Conclusions

Many models of fluorescence microscopes use the monopole and scalar approximations, but complete models need to consider dipole and vector optics effects. In this work we have introduced several transfer functions that simplify the mapping between the dipole density and the irradiance pattern on the detector. In future papers of this series we will calculate these transfer functions for specific instruments and use the results to simulate and analyze data collected by these instruments.

Funding

National Institute of Health (NIH) (R01GM114274, R01EB017293).

Acknowledgments

We thank Kyle Myers, Harrison Barrett, Scott Carney, Luke Pfister, Jerome Mertz, Sjoerd Stallinga, Mikael Backlund, Matthew Lew, Min Guo, Yicong Wu, Shalin Mehta, Abhishek Kumar, Peter Basser, Marc Levoy, Michael Broxton, Gordon Wetzstein, Hayato Ikoma, Laura Waller, Ren Ng, Tomomi Tani, Michael Shribak, Mai Tran, Amitabh Verma, Xiaochuan Pan, Emil Sidky, Chien-Min Kao, Phillip Vargas, Dimple Modgil, Sean Rose, Corey Smith, Scott Trinkle, and Jianhua Gong for valuable discussions during the development of this work. TC was supported by a University of Chicago Biological Sciences Division Graduate Fellowship, and PL was supported by a Marine Biological Laboratory Whitman Center Fellowship. Support for this work was provided by the Intramural Research Programs of the National Institute of Biomedical Imaging and Bioengineering.

Disclosures

The authors declare that there are no conflicts of interest related to this article.

Appendix A Spherical harmonics and the spherical Fourier transform

The spherical harmonic function of degree $\ell$ and order $-\ell\leq m\leq\ell$ is defined as [32]

[TABLE]

where $P_{\ell}^{m}(\cos\vartheta)$ are the associated Legendre polynomials with the Condon-Shortley phase

[TABLE]

and $P_{\ell}(x)$ are the Legendre polynomials defined by the recurrence

[TABLE]

The spherical harmonics are orthonormal, which means that

[TABLE]

where $\delta_{\ell\ell^{\prime}}$ denotes the Kronecker delta. The spherical harmonics form a complete basis, so an arbitrary function on the sphere $f(\mathbf{\hat{s}})$ can be expanded into a sum of weighted spherical harmonic functions

[TABLE]

We can find the spherical harmonic coefficients $F_{\ell}^{m}$ for a given function using Fourier’s trick—multiply both sides by $Y_{\ell}^{m*}(\mathbf{\hat{s}})$, integrate over the sphere, and exploit orthogonality to find that

[TABLE]

The coefficients $F_{\ell}^{m}$ are called the spherical Fourier transform of a spherical function.