Combining nonparametric spatial context priors with nonparametric shape priors for dendritic spine segmentation in 2-photon microscopy images

TL;DR

This paper introduces a segmentation method that combines nonparametric spatial context priors with shape priors and a learned data term, significantly improving dendritic spine segmentation in microscopy images.

Contribution

It proposes a novel approach integrating nonparametric spatial context priors with shape and intensity-based data terms for improved segmentation.

Findings

Significant improvement in dendritic spine segmentation accuracy.

Effective handling of overlapping pixel intensity distributions.

Enhanced segmentation robustness in 2D and 3D microscopy images.

Abstract

Data driven segmentation is an important initial step of shape prior-based segmentation methods since it is assumed that the data term brings a curve to a plausible level so that shape and data terms can then work together to produce better segmentations. When purely data driven segmentation produces poor results, the final segmentation is generally affected adversely. One challenge faced by many existing data terms is due to the fact that they consider only pixel intensities to decide whether to assign a pixel to the foreground or to the background region. When the distributions of the foreground and background pixel intensities have significant overlap, such data terms become ineffective, as they produce uncertain results for many pixels in a test image. In such cases, using prior information about the spatial context of the object to be segmented together with the data term can bring a curve to a plausible stage, which would then serve as a good initial point to launch shape-based segmentation. In this paper, we propose a new segmentation approach that combines nonparametric context priors with a learned-intensity-based data term and nonparametric shape priors. We perform experiments for dendritic spine segmentation in both 2D and 3D 2-photon microscopy images. The experimental results demonstrate that using spatial context priors leads to significant improvements.

Figures

Tables

Table 1. Table 1 : Average Dice score results on both 2 D 𝐷 D and 3 D 𝐷 D dendritic spine data sets

	Proposed method	Context priors with learned-intensity-based data term	[9]	[3]	[7]
3$D$ Dendritic spine data set	0.7022	0.6597	0.0989	0.4319	0.5857
2$D$ Dendritic spine data set	0.8590	0.8341	0.5188	0.5759	0.7347

Equations

$$p(c|y,l)\propto p(y,l|c)p(c)\propto p(y|c)p(l|c)p(c).$$

$$E(c)=-\log p(y|c)-\log p(l|c)-\log p(c).$$

$$E(\varphi (c))=-\log p(y|\varphi (c))-\log p(l|\varphi (c))-\log p(\varphi (c)).$$

$$-\log p(y|\varphi (c))=-{\int }_{\varphi (c)<0}{p}_{fg}(y(x))\mathrm{d}x-{\int }_{\varphi (c)>0}{p}_{bg}(y(x))\mathrm{d}x$$

$$p_{fg}(y(x))=\frac{1}{|I_{fg}|}\sum _{i=1}^{|I_{fg}|}\mathcal{N}(y(x);I_{fg}(i),\sigma )$$

$$p_{bg}(y(x))=\frac{1}{|I_{bg}|}\sum _{i=1}^{|I_{bg}|}\mathcal{N}(y(x);I_{bg}(i),\sigma ).$$

$$-\log p(l|\varphi (c))=-{\int }_{\varphi (c)<0}{q}_{fg}(x)\mathrm{d}x-{\int }_{\varphi (c)>0}{q}_{bg}(x)\mathrm{d}x$$

$$q_{fg}(x)=\frac{1}{|L_{fg}|}\sum _{i=1}^{|L_{fg}|}\mathcal{N}(x;L_{fg}(i),\sigma )$$

$$q_{bg}(x)=\frac{1}{|L_{bg}|}\sum _{i=1}^{|L_{bg}|}\mathcal{N}(x;L_{bg}(i),\sigma )$$

$$\frac{-\partial \log p(y(x)|\varphi (c))}{\partial \varphi (c)}=\log \frac{p_{bg}(y(x))}{p_{fg}(y(x))},$$

$$\frac{-\partial \log p(l(x)|\varphi (c))}{\partial \varphi (c)}=\log \frac{q_{bg}(x)}{q_{fg}(x)},$$

$$\frac{\partial -\log p(\varphi (c))}{\partial \varphi (c)}=\frac{1}{p(\varphi (c))}\frac{1}{{\sigma }^2}\frac{1}{n}\sum _{i=1}^n\mathcal{N}(\varphi (c);\varphi (c_i),\sigma I)\times (\varphi (c_i)-\varphi (c))$$

Full text

**Index Terms— ** Nonparametric shape priors, spatial context priors, spine segmentation, 2-photon microscopy.

1 Introduction

Segmentation of images involving limited and low quality data is a challenging problem and requires prior information about the shapes of the objects to be segmented for accurate results [1, 2, 3]. Earlier work involved the use of curve-length penalties, essentially providing simple prior information about shape regularity [4]. Later, by using linear analysis tools such as principal component analysis (PCA), more informative shape priors learned from training samples have been incorporated into the segmentation process [5]. However, these methods can only handle Gaussian-like, unimodal, shape prior densities. In order to handle multimodal shape densities, methods that exploit nonparametric shape priors have been proposed [1, 2, 3, 6]. The major deficiency of these techniques is that they often combine such priors with simplistic data terms. A common underlying assumption of such data terms is that the foreground and the background regions in the image are homogeneous, e.g., intensities are piecewise constant or piecewise smooth [7]. Learning-based intensity distributions have been integrated as data terms with nonparametric shape priors to handle more complicated intensity distributions [7].

Integration of nonparametric shape priors with learned-intensity-based data terms has led to improved segmentation results when the foreground and background intensities are not homogeneous [7]. Data driven segmentation is an important initial step of segmentation methods that exploit shape priors. In learned-intensity-based data terms, intensity distributions of the foreground and background regions are learned from a training set, possibly in a nonparametric fashion. In challenging segmentation tasks, the foreground and background intensity distributions have significant overlap, causing the segmentation algorithm to produce uncertain region assignments for many pixels, leading to poor segmentation results.

Let us consider the 2-photon microscopy image of a dendritic spine in Figure 1(a) as an illustrative example. Dendritic spines are small protrusions from a neuron’s dendrite, and are of interest in neuroscience research as their density and morphology are related to several functions including learning and memory. For analysis of their morphology, their segmentation is crucial. The probability density functions (pdfs) learned from a training set for the foreground and background intensities are shown in Figure 1(h). Note that bright intensities might appear with almost equal probabilities in both the foreground and background regions according to the pdfs. Since data-driven segmentation is performed based on these intensity distributions, bright pixels from the foreground region may easily be assigned to the background region and vice versa. Segmentation of the dendritic spine image using a learned-intensity-based data term produces the segmentation result shown in Figure 1(d). Note that some bright pixels are assigned to foreground region, while they should be in the background. Spatial context priors incorporate higher level of information about the pixel locations of the objects to be segmented. Therefore, by using spatial context priors (as described later in this paper) together with learned-intensity-based data term, we exploit information about the location of pixels in the image together with the intensity pdfs. This produces the segmentation result shown in Figure 1(e). The result obtained by using context priors together with learned-intensity-based data term is closer to the ground truth (see Figure 1(b)) than the one obtained by purely data-driven segmentation. Therefore, the segmentation in Figure 1(e) would be a better initialization for shape-based segmentation. Giving the boundary in Figure 1(d) as initialization to an algorithm using a learned-intensity-based data term and shape priors produces the result in Figure 1(f) which is not quite similar to the ground truth. On the other hand. the approach developed in this paper, exploiting both spatial context and shape priors, initialized by the boundary in Figure 1(e) produces a closer segmentation result to the ground truth as shown in Figure 1(g).

In this paper, we propose a new segmentation approach that uses spatial context priors together with shape priors and a learned-intensity-based data term in a Bayesian framework. We learn all the densities involved in a nonparametric fashion from a training set. To the best of our knowledge such spatial context priors have not been used together with learned-intensity-based data terms and nonparametric shape priors for segmentation in the literature. We perform experiments for dendritic spine segmentation on both 2$D$ and 3$D$ 2-photon microscopy images and demonstrate the improvements provided by the proposed approach.

This approach would be of potential use in challenging biomedical and biological image segmentation problems (e.g. brain tumor segmentation, prostate segmentation, etc.) in which one can collect statistical information regarding the shapes, intensities, and locations of the particular structures of interest, from training data. Our framework provides a principled probabilistic formulation and an associated algorithm to exploit such information.

2 Proposed method

Let us assume that we have a training set of $n$ aligned intensity images $\mathcal{Y}=\{y_{1},\cdots,y_{n}\}$ and their corresponding manual segmentations, $\mathcal{C}=\{c_{1},\cdots,c_{n}\}$. Given $\mathcal{Y}$ and $\mathcal{C}$, we can construct multisets111We use multiset to indicate a set that contains repeated values., $I_{fg}$ and $I_{bg}$, that store all intensity values for pixels in the training set that are located in the foreground and background regions, respectively. Similarly, we can construct multisets, $L_{fg}$ and $L_{bg}$, that store the locations of all pixels in the foreground and background regions, respectively. We define the posterior probability density function of segmenting curve $c$ given observed image $y$ and spatial context of the object to be segmented, $l$ as

[TABLE]

where $y$ and $l$ contain the pixel intensities and the associated locations. Hence our formulation exploits information, gathered from from training data, not only about intensities of pixels in different regions but also about locations of pixels belonging to different regions in a principled probabilistic framework.

By taking the negative logarithm of Equation (1), we can define the following energy function to be minimized for segmentation:

[TABLE]

In this paper, we use level sets as shape representation. Level set representation is essentially a mapping, $\phi$ from the binary space to the real space. In the literature, it has been found more convenient to work with level sets to handle topological shape changes [8] and computing gradients [3]. In the level set representation we use, values less than zero indicate foreground region whereas values greater than zero indicate background region. Using level set representation, the energy function in Equation (2) becomes

[TABLE]

We define $-\log p(y|\phi(c))$ as proposed in [7]:

[TABLE]

where

[TABLE]

and

[TABLE]

In Equations (5) and (6), $\mathcal{N}(.;\mu,\sigma)$ indicates a Gaussian density with mean $\mu$ and standard deviation $\sigma$ and $I_{fg}(i)$ ($I_{bg}(i)$) indicates $i^{th}$ element of $I_{fg}$ ($I_{bg}$). Hence (5) and (6) provide nonparametric pdf estimates of intensities in the foreground and background regions.

Similarly, we define $-\log p(l|\phi(c))$ as:

[TABLE]

where

[TABLE]

and

[TABLE]

Finally, we define $p(\phi(c))$, the shape prior density, as $p(\phi(c))=\frac{1}{n}\sum_{i=1}^{n}\mathcal{N}(\phi(c);\phi(c_{i}),\sigma I)$ where $\mathcal{N}(.;\mu,\sigma I)$ indicates a Gaussian density with mean vector $\mu$ and covariance matrix $\sigma I$, and $I$ is the identity matrix.

The segmentation problem turns into the problem of finding a boundary $c$ that minimizes the energy functional in Equation (2). To achieve this, we minimize Equation (3) using gradient descent which requires computing the partial derivative of $E(\phi(c))$ with respect to $\phi(c)$. This is equivalent to computing partial derivatives of each component involved which are written as

[TABLE]

[TABLE]

and

[TABLE]

Given partial derivatives of each term, we perform curve evolution in the gradient direction of these terms to find the desired segmentation. The algorithmic steps of the proposed segmentation approach are: (i) Perform curve evolution until convergence using the learned-intensity-based data term and the context priors terms, i.e., (10) and (11); (ii) Take the curve found in (i) as initialization and perform curve evolution until convergence using the learned-intensity-based data term together with the spatial context and shape priors terms, i.e., (10), (11), (12).

3 Experimental results

In this section, we present experimental results of the proposed approach on the dendritic spine segmentation problem. We perform experiments on both 2$D$ and 3$D$ dendritic spine data sets [1]. The dendritic spine data sets are obtained from the Neuronal Structure and Function Laboratory of the Champalimaud Neuroscience Foundation, Lisbon.

Both 2$D$ and 3$D$ dendritic spine data sets contain 30 intensity images with manual segmentations. We perform experiments on these data sets on a leave-one-out basis; one image for test and remaining 29 for training. We compare the performance of the proposed approach which uses the learned-intensity-based data term, as well as spatial context and shape priors with the approaches proposed in [9], [3], and [7] in terms of Dice score [10]. We also obtain Dice score results for the segmentations obtained by using spatial context priors with the learned-intensity-based data term which we use as the initial step of our proposed approach as discussed before. Note that Dice score takes values between 0 and 1 where higher is better.

The average Dice score results of 30 spines obtained from the experiments on the 2$D$ and 3$D$ dendritic spine data sets are shown in Table 1. We first discuss the 3$D$ results. The method in [9] does not use any shape prior information; it just uses a data term that assumes the foreground and the background intensities are homogeneous. Therefore, this method produces the worst results among all methods that we use in our evaluations. The method in [3] uses a nonparametric shape prior term together with the data term in [9]. Although, this leads to some improvement over the results of the method in [9], the results of [3] are still around $0.4319$ for the 3$D$ case in terms of Dice score. The method in [7] uses the learned-intensity-based data term together with the nonparametric shape priors term in [3]. This slightly improves the average Dice score result of [3]. Among all approaches, the proposed approach which uses the learned-intensity-based data term, as well as spatial context and shape priors produces segmentations with the highest Dice score results averaged over 30 spine images. Performing segmentation using spatial context priors together with the learned intensity-based data term (the proposed approach without shape priors) produces better results than [9], [3], and [7], demonstrating the positive impact of the context prior term developed in this paper. Finally, we performed a statistical significance analysis comparing the Dice score results of the proposed approach and each of the competing methods using a t-test. The statistical tests demonstrate that the difference between the Dice score results of the proposed and other methods is statistically significant at the $5\%$ significance level for the 3D dendritic spine segmentation experiment.

The relative performance of the methods on the 2$D$ dendritic spine data set is very similar to the 3$D$ case in terms of Dice score as shown in Table 1. The proposed approach achieves the best results in terms of Dice score averaged over 30 spine images. As in the 3$D$ case, Dice score results obtained by using context priors with the learned-intensity-based data term are better than those of the existing methods. The results obtained by methods in [9] and [3] produce the lowest and the second lowest Dice score results, respectively. Finally, average Dice score achieved by using the method in [7] is around 0.7347 which is lower than the methods that use context priors and higher than the other two methods. We performed statistical significance analysis using a t-test on the Dice score results of the 2$D$ segmentation experiment as well. Statistical significance analysis results demonstrate that the Dice score results of the proposed approach and the methods in [9], [3], and [7] are statistically significant at the $5\%$ significance level. However, there is no statistically significant difference between the Dice score results of the proposed approach and the approach that uses spatial context priors and the learned-intensity-based data term. Some visual segmentation results obtained by all these approaches are shown in Figure 2. Note that visual segmentation results support our quantitative analysis.

4 Conclusion

We have proposed a segmentation approach that uses spatial context priors together with shape priors and a learned-intensity-based data term. This is a principled probabilistic approach for incorporation of information about the likely locations of particular regions in the image, within the context of a shape-prior-based segmentation framework. Assuming such statistical information about spatial context can be extracted from training data, this approach would particularly be of value in challenging segmentation problems which exhibit region-based intensity distributions with significant overlaps, making the problem harder. In this paper we have demonstrated the benefits provided by this approach in the context of microscopic neuroimage analysis, in particular on the problem of dendritic spine segmentation. However we believe the approach can also be useful in other segmentation problems, especially in biomedical or biological imaging.