Adaptive Morphological Reconstruction for Seeded Image Segmentation

TL;DR

This paper introduces an adaptive morphological reconstruction method that improves seeded image segmentation by effectively filtering seeds, being scale-insensitive, and enabling hierarchical segmentation, resulting in better accuracy and efficiency.

Contribution

The paper proposes a novel adaptive morphological reconstruction technique that enhances seed filtering, scale insensitivity, and hierarchical segmentation in seeded image segmentation algorithms.

Findings

AMR improves seed filtering accuracy.

AMR enhances segmentation quality over state-of-the-art methods.

AMR reduces computational time in segmentation tasks.

Abstract

Morphological reconstruction (MR) is often employed by seeded image segmentation algorithms such as watershed transform and power watershed as it is able to filter seeds (regional minima) to reduce over-segmentation. However, MR might mistakenly filter meaningful seeds that are required for generating accurate segmentation and it is also sensitive to the scale because a single-scale structuring element is employed. In this paper, a novel adaptive morphological reconstruction (AMR) operation is proposed that has three advantages. Firstly, AMR can adaptively filter useless seeds while preserving meaningful ones. Secondly, AMR is insensitive to the scale of structuring elements because multiscale structuring elements are employed. Finally, AMR has two attractive properties: monotonic increasingness and convergence that help seeded segmentation algorithms to achieve a hierarchical segmentation. Experiments clearly demonstrate that AMR is useful for improving algorithms of seeded image segmentation and seed-based spectral segmentation. Compared to several state-of-the-art algorithms, the proposed algorithms provide better segmentation results requiring less computing time. Source code is available at https://github.com/SUST-reynole/AMR.

Figures

Equations

$$R_g^{\delta }(f)={\delta }_g^{(n)}(f),$$

$$R_g^{\epsilon }(f)={\epsilon }_g^{(n)}(f),$$

$$\{\begin{array}{r}{R}_g^{\gamma }(f)=R_g^{\delta }(R_g^{\epsilon }(f))\\ R_g^{\varphi }(f)=R_g^{\epsilon }(R_g^{\delta }(f))\end{array}.$$

$$y=\frac{1}{\lambda }\sum _{j=1}^{\lambda }{g}_j,$$

$$y=\frac{1}{\lambda }\sum _{j=1}^{\lambda }{\omega }_j{g}_j,$$

$$\psi (g,s,m)={\vee }_{s\le i\le m}\left\{R_g^{\varphi }(f{)}_{b_i}\right\}.$$

$$p\le q\Rightarrow \psi (g,s,p)\le \psi (g,s,q).$$

$$H_z=H_0,ifz\le 0,$$

$$\exists n\in N^{+},H_z=\left\{\Omega \right\},\forall z\ge n,$$

$$p\le q\Rightarrow H_p\subseteq H_q,1\le p,q<n,$$

$$\psi (g,s,p)\le \psi (g,s,q)\Rightarrow S(\psi (g,s,p))\subseteq S(\psi (g,s,q)),$$

$$H_0\subseteq H_1\subseteq \cdots \subseteq H_m.$$

$$\psi (g,s,m)=\psi (g,s,m+j),$$

$$J(g,s)=max|\psi (g,s,m)-\psi (g,s,m-1)|,$$

$$f={\delta }_{b_m}\left(g\right),m\to \infty ,\underset{m\to \infty }{\lim }{\delta }_{b_m}\left(g\right)=\max \left(g\right)$$

$$f=max(g),$$

$$\begin{array}{rl}{\epsilon }_g^{(1)}(f)& =\epsilon (f)\vee g\\ & =\epsilon (max(g))\vee g\\ & =max(g),\end{array}$$

$$\begin{array}{rl}{\epsilon }_g^{(n)}(f)& =\epsilon ({\epsilon }_g^{(n-1)}(f))\vee g\\ & =max(g).\end{array}$$

$$\begin{array}{r}{R}_g^{(\epsilon )}(f)=max(g).\end{array}$$

$$\begin{array}{rl}\underset{m\to \infty }{\lim }{R}_g^{\gamma }(f{)}_{b_m}& =\underset{m\to \infty }{\lim }{R}_g^{\delta }(R_g^{\epsilon }(f))\\ & =R_g^{\delta }(max(g))\\ & =max(g).\end{array}$$

$$\begin{array}{r}\underset{m\to \infty }{\lim }{R}_g^{\varphi }(f{)}_{b_m}=min(g).\end{array}$$

$$p\le q\Rightarrow \psi (g,s,p)\le \psi (g,s,q)$$

$$\psi (g,s,p)=\vee \left\{R_g^{\varphi }(f{)}_{b_s},R_g^{\varphi }(f{)}_{b_{s+1}},\cdots ,R_g^{\varphi }(f{)}_p\right\},$$

$$\psi (g,s,p)=\vee \left\{R_g^{\varphi }(f{)}_{b_s},R_g^{\varphi }(f{)}_{b_{s+1}},\cdots ,R_g^{\varphi }(f{)}_q\right\}.$$

$$\psi (g,s,p)=\vee \left\{\psi (g,s,p),R_g^{\varphi }(f{)}_{b_{p+1}},\cdots ,R_g^{\varphi }(f{)}_q\right\},$$

$$\psi (g,s,p)\le \psi (g,s,q).$$

$$\begin{array}{rl}\psi (g,s,m)& =\psi (g,s,m+j),\\ min(\psi (g,s,m))& \ge max(R_g^{\varphi }(f{)}_{b_{m+1}})\end{array}$$

$$\underset{m\to \infty }{\lim }\psi (g,s,m)={\vee }_{s\le i\le m}\left\{R_g^{\varphi }(f{)}_{b_i}\right\}$$

$$\begin{array}{rl}& ={\vee }_{s\le i\le m}\left\{R_g^{\varphi }(f{)}_{b_i}\right\}\vee \left\{R_g^{\varphi }(f{)}_{b_{m+1}},R_g^{\varphi }(f{)}_{b_{m+2}},\cdots ,R_g^{\varphi }(f{)}_{b_{\infty }}\right\}\\ & =\psi (g,s,m)\vee \left\{R_g^{\varphi }(f{)}_{b_{m+1}},R_g^{\varphi }(f{)}_{b_{m+2}},\cdots ,R_g^{\varphi }(f{)}_{b_{\infty }}\right\}\end{array}$$

$$max(R_g^{\varphi }(f{)}_{b_{m+1}})\ge max(R_g^{\varphi }(f{)}_{b_{m+2}})\ge \cdots \ge R_g^{\varphi }(f{)}_{b_{\infty }}.$$

Full text

I Introduction

Morphological reconstruction (MR) [1] is a powerful operation in mathematical morphology. It has been widely used in image filtering [2], image segmentation [3], and feature extraction [4], etc. Among these applications, one of the most important applications is that MR is often used in seeded segmentation algorithms [5], [6] such as watershed transformation (WT) [7] and power watershed (PW) [8] to reduce over-segmentation caused by image noise and details. However, there are two drawbacks [9], [10] when MR is used in seeded segmentation algorithms.

• •

It is difficult to reduce over-segmentation while obtaining a high segmentation accuracy for seeded segmentation algorithms (we use MR-WT to denote MR-based watershed transform and use MR-PW to denote MR-based power watershed). Although MR is able to filter noise in gradient images, some important contour details are smoothed out as well.

• •

MR is sensitive to the scale of structuring elements. In practical applications, if the scale is too small, the reconstructed gradient image suffers from a serious over-segmentation. Oppositely, if the scale is too large, the reconstructed gradient image suffers from an under-segmentation.

Generally, MR is used in watershed transform to improve the segmentation effect by employing a structuring element to filter regional minima [11]. However, it is very difficult to filter useless regional minima while preserving meaningful ones by simply considering one single-scale structuring element. Although $H$-min imposition [12] is a simple and efficient method for over-segmentation reduction, it relies on a threshold choice and is likely to miss some important boundaries. Region merging [13], [14] is also a popular method for this, but it requires iterating and renewing edge weight leading to a high computing burden. In addition, some researchers employ reasonable contour detection methods, e.g., globalized probability of boundary (gPb) [15] that combines the multiscale information from brightness, color and texture, to achieve better image segmentation. However, the gPb is computationally expensive because it combines too many feature cues for contour detection. To speed up the algorithm of contour detection, Dollar and Zitnick [16] took the advantage of the structure present in regional image patches and random decision forests, and proposed a fast structured edge (SE) detection approach using structured forests. This algorithm obtains real-time performance and state-of-the-art edge detection but requires a huge amount of memory for training data. To reduce memory requirement, Hallman and Fowlkes [17] proposed a simple and efficient model to learn contour detection, namely oriented edge forests (OEF). Although these improved contour detectors are superior to traditional detectors, e.g. Sobel or Canny, and they are helpful for improving subsequent image segmentation, they still generate a large number of seeds leading to serious over-segmentations.

In practice, contour detection methods are usually combined with other approaches to improve image segmentation effect. For example, Fu et al. [18] proposed a robust image segmentation approach using contour-guided color palettes by integrating contour and color cues, where SE, mean-shift algorithm [19], region merging, and spectral clustering [20] are combined to achieve better segmentation results. However, the approach is complex because it combines several different algorithms that requires many parameters.

In this paper, we propose an adaptive morphological reconstruction (AMR) operation that is able to generate a better seed image than MR to improve seeded segmentation algorithms. Firstly, AMR employs multiscale structuring elements to reconstruct a gradient image. Secondly, a pointwise maximum operation on these reconstructed gradient images is performed to obtain the final adaptive reconstruction result. Because AMR employs small structuring elements to reconstruct pixels of large gradient magnitudes while employing large structuring elements to reconstruct pixels of small gradient magnitudes in a gradient image, AMR is able to obtain better seed images to improve the seeded segmentation algorithms. Our main contributions are summarized as follows.

• •

Multiscale structuring elements are employed by AMR, and different scaled structuring elements are adaptively adopted by pixels of different gradient magnitudes without computing the local features of a gradient image.

• •

AMR has a convergence property and a monotonic increasing property, the two properties help seeded segmentation algorithms to achieve a hierarchical segmentation.

• •

AMR has a low computational complexity, and it can help seed-based spectral segmentation to achieve better image segmentation results than the-state-of-art algorithms.

The rest of the paper is organized as follows. In the next section, the research background related with AMR is introduced and analyzed. On this basis, AMR is proposed, and its two properties, monotonic increasingness and convergence are carefully analyzed in Section III. To demonstrate the superiority of AMR, AMR is used for seeded image segmentation and seed-based spectral segmentation. Experiments are presented in Section IV, followed by the conclusion in Section V.

II Background

II-A Morphological Reconstruction

MR is an image transformation that requires two input images, a marker image and a mask image. Let two grayscale images $f$ and $g$ denote the marker image that is the starting point for the transformation and the mask image that constrains the transformation, respectively [21]. If $f\leq g$, which means $f$ is pointwise less than or equal to $g$, the morphological dilation reconstruction $(R^{\delta})$ of $g$ from $f$ is denoted by

[TABLE]

where $\delta_{g}^{(1)}(f)=\delta(f)\wedge g$, $\delta_{g}^{(k)}(f)=\delta(\delta_{g}^{(k-1)}(f))\wedge g$ for $2\leq k\leq n$, $k,n\in N^{+}$ satisfies $\delta_{g}^{(n)}(f)=\delta_{g}^{(n-1)}(f)$. The symbol $\delta$ represents the elementary morphological dilation operation, and $\wedge$ stands for the pointwise minimum at each pixel of two images as shown in Fig. 1(a).

Similarly, if $f\geq g$, the morphological erosion reconstruction $(R^{\varepsilon})$ of $g$ from $f$, which is the dual operation of $R^{\delta}$, is defined as

[TABLE]

where $\varepsilon_{g}^{(1)}(f)=\varepsilon(f)\vee g$, $\varepsilon_{g}^{(k)}(f)=\varepsilon(\varepsilon_{g}^{(k-1)}(f))\vee g$ for $2\leq k\leq n$, $k,n\in N^{+}$ satisfies $\varepsilon_{g}^{(n)}(f)=\varepsilon_{g}^{(n-1)}(f)$. The symbol $\varepsilon$ represents the elementary morphological erosion operation, and $\vee$ stands for the pointwise maximum at per pixel of two images as shown in Fig. 1(b).

To further illustrate the principle of MR for image transformation, we present an example for the binary MR as shown in Fig. 2, where the red regions denote seeds, i.e., the marker image $f$.

According to Fig. 2 and (1)-(2), a suitable marker image is important for MR. We have known that $f\leq g$ for $R^{\delta}$ while $f\geq g$ for $R^{\varepsilon}$. Thus, there are lots of choices for $f$. Different marker images corresponds to different reconstruction results as shown in Fig. 3. To obtain an efficient $f$ in practice, the marker image is usually obtained by performing a transformation on the corresponding mask image [22]-[24]. For example, the erosion or dilation result of a mask image is often considered as a marker image [25], i.e., $f=\varepsilon_{b_{i}}(g)$ or $f=\delta_{b_{i}}(g)$, where $b_{i}$ is a disk shaped structuring element, the radius of $b_{i}$ is $i$, $i\in N^{+}$. Therefore, MR is sensitive to the parameter $i$ because the marker image is decided by the scale of the structuring element.

As compositional morphological opening and closing operations show better performance than elementary morphological erosion and dilation operations for image filtering, feature extraction, etc., we present the definition of compositional morphological opening and closing reconstructions ($R^{\gamma}$ and $R^{\phi}$) of $g$ from $f$ as follows

[TABLE]

II-B Multiscale and Adaptive Mathematical Morphology

For image filtering and enhancement using morphological operators, a large-scale structuring element can suppress noise but may also blur the image details, whereas a small-scale structuring element can preserve image details but may fail to suppress noise. Some researchers proposed multiscale and adaptive morphological operators to improve the performance of traditional morphological operators. However, most multiscale morphological operators [26], [27] such as morphological gradient operators and morphological filtering operators, average all scales of morphological operation results as final output

[TABLE]

where $y$ is the final output result, $j$ is the radius of the structuring element, $1\leq j\leq\lambda$, $j$, $\lambda\in N^{+}$. Although the average result is superior to the result based on single-scale morphological operators, it causes contour offset and mistakes. Some researchers improved multiscale morphological operators by introducing a weighted coefficient to (4), and they defined adaptive multiscale morphological operators as follows [28]

[TABLE]

where $w_{j}$ is the weighted coefficient on the $j$th scale result. However, because the computing of weighted coefficients is complex, the adaptive multiscale morphological operators have a low computational efficiency. Moreover, the weighted average result is similar to average result because it is difficult to obtain the optimal weighted coefficient, even though the former is slightly better than the latter.

Although lots of adaptive multiscale morphological operators [29]-[32] have been proposed, it can be seen from (4)-(5) that both the multiscale and adaptive morphological operators employ a linear combination of different-scale results to improve single-scale morphological gradient or filtering operators. Because the linear combination is unsuitable for multiscale morphological reconstruction operation, in this paper, we try to employ a non-linear combination (i.e., the pointwise maximum operation denoted by $\vee$) to design adaptive morphological reconstruction operators. These operators are different from conventional multiscale and adaptive morphological operators employing linear combination in (4)-(5). We use non-linear operation $\vee$ instead of linear combination since the former is more suitable than the later for the removal of useless seeds in seeded image segmentation.

II-C Seeded Segmentation

Seeded segmentation algorithms, such as graph cuts [33], random walker [34], watersheds [7], and power watershed [8] have been widely used in complex image segmentation tasks due to their good performance [35]. It is not required to give seed images for both graph cuts and random walker because they usually consider each pixel as a seed. However, a seed image is necessary for WT and PW by computing the regional minima of a gradient image.

Since both WT and PW obtain seeds from a gradient image that often includes a huge number of seeds generated by noise and unimportant texture details, they usually suffer from over-segmentation. A larger number of approaches for addressing over-segmentation was proposed, and these approaches can be categorized into two groups.

• •

Feature extraction or feature learning is used to obtain a better gradient image that enhances important contours while smoothing noise and texture details [15]-[17].

• •

MR is used for gradient reconstruction to reduce the number of regional minima [36]-[38].

For the first group of approaches, gPb, OEF, and SE are popular for reducing over-segmentation as shown in Figs. 4-5. In Fig. 5, although gPb, OEF, and SE provide better gradient images that can reduce over-segmentation for WT and PW, the segmentation results are still poor compared to ground truths shown in Fig. 4.

The second group of approaches depends on MR and WT, it is denoted by MR-WT. Najman and Schmitt [36] employed MR to remove regional minima to reduce over-segmentation. Furthermore, a dynamic threshold is used to change the gradient magnitude that is smaller than the threshold, and then a hierarchical segmentation result is obtained. Wang [26] proposed a multiscale morphological gradient algorithm (MMG) for image segmentation using watersheds. The proposed MMG employs multiple structuring elements to obtain a better gradient image, and uses MR to remove regional minima to improve watershed segmentation.

Fig. 6 illustrates the principle of a seeded segmentation framework based on MR-WT. We can see that the number of the regional minima in the gradient image decreases rapidly with the increase of $i$, but the boundary is also destroyed simultaneously. It is clear that the larger structuring element corresponds to fewer seeds. One major reason is that MR employs a single-scale structuring element, which equally treats all pixels of different gradient magnitudes in the gradient image. For example, in dilation reconstruction, the marker image $f=\varepsilon_{b_{i}}(g)$ converges to the minimum grayscale value of pixels in the mask image as the value of $i$ increases. Obviously, both large and small structuring elements lead to poor reconstruction results while a moderate-sized structuring element achieves a rough balance via sacrificing contour precision. Therefore, it is difficult to obtain a good seed image by employing a single-scale structuring element. Although many researchers employ multiscale structuring elements to generate a better gradient image, there are few studies on multiscale MR for gradient images. Moreover, the fusion of different-scale results is also a problem.

II-D Spectral Segmentation

It is well-known that spectral clustering [20] is greatly successful due to the fact that it does not make strong assumptions on data distribution, and it is implemented efficiently even for large datasets, as long as we make sure that the affinity matrix is sparse. However, since the size of the affinity matrix is $(M\times N)^{2}$ for an image of size $M\times N$, and it is not sparse because of Gaussian similarity measure, spectral clustering is often inefficient for image segmentation due to eigenvalue decomposition of the huge affinity matrix. To address the issue, a great number of algorithms have been proposed to construct a smaller affinity matrix and thus to improve the computational efficiency of spectral clustering [39]-[42]. Most of these algorithms employ pre-segmentation (superpixel) methods such as the simple linear iterative clustering (SLIC) [43], mean-shift [19], linear spectral clustering (LSC) [44], and superpixel hierarchy [45], to reduce the number of pixels of the original image and, in turn, reduces the size of the affinity matrix. As an example, Zhang et al. [46] proposed a fast image segmentation approach that is a re-examination of spectral clustering on image segmentation. The approach provides better image segmentation results yet requires a long running time.

The popular superpixel approaches have some drawbacks for spectral segmentation. Firstly, mean-shift algorithm involves three parameters and it is sensitive to these parameters. Secondly, SLIC only generates superpixels that include regular regions, and these regions have a similar shape and size. Finally, LSC is superior to SLIC because LSC successfully connects a local feature with a global optimization objective function, so that LSC can generate more reasonable segmentation results. However, similar to SLIC, LSC also provides superpixels that include regular regions with a similar shape and size.

As seed-based spectral segmentation algorithms are sensitive to pre-segmentation results, an excellent pre-segmentation algorithm can improve segmentation results generated by seed-based spectral segmentation algorithms.

III Adaptive morphological reconstruction

III-A The Proposed AMR

To overcome the drawback of MR on regional minima filtering, we propose an AMR that is able to filter useless regional minima and maintains meaningful ones generated by salient objects. Fig. 7 shows the motivation of AMR in which multiscale structuring elements are employed to reconstruct a gradient image, i.e., small structuring elements are adopted by pixels of large gradient magnitude while large structuring elements are adopted by pixels of small gradient magnitude.

Definition 1. Let $b_{s}\subseteq\cdots b_{i}\subseteq b_{i+1}\cdots\subseteq b_{m}$ be a series of nested structuring elements, where $i$ is the scale parameter of a structuring element, $1\leq s\leq i\leq m$, $s,i,m\in N^{+}$. For a gradient image $g$ such that $f=\varepsilon_{b_{i}}(g)$ and $f\leq g$, the adaptive morphological reconstruction denoted by $\psi$ of $g$ from $f$ is defined as

[TABLE]

Note that the pointwise maximum operation is only suitable for $R^{\phi}$, but not suitable for $R^{\gamma}$. Because $\lim\limits_{m\rightarrow\infty}R_{g}^{\gamma}(f)_{b_{m}}=max(g)$ (the proof is presented in Appendix A) and $\lim\limits_{m\rightarrow\infty}\vee_{s\leq i\leq m}\left\{R_{g}^{\gamma}(f)_{b_{i}}\right\}=max(g)$, $\psi(g,s,m)$ is unable to obtain a significantly convergent gradient image if $\psi(g,s,m)=\vee_{s\leq i\leq m}\left\{R_{g}^{\gamma}(f)_{b_{i}}\right\}$.

We apply AMR to the gradient image shown in Fig. 6. The reconstruction and segmentation results are shown in Fig. 8, where the adopted structuring elements are disk and $s=1$. More detailed comparisons are shown in Fig. 9. By comparing Fig. 8 with Fig. 6, it is obvious that AMR obtains better seed images than MR due to the fact that the non-linear operation $\vee$ is able to remove efficiently useless seeds.

To further show the influence of $s$ on AMR, Fig. 10 shows segmentation results provided by AMR through changing the value of $s$. We can see that there are some small segmented areas when the value of $s$ is small. These small areas are merged by increasing the value of $s$. However, although a large $s$ leads to the merge of small areas, the precision of object contours will be decreased as shown in Fig. 6. Therefore, we usually set $1\leq s\leq 3$ for a moderate-sized image.

III-B The Monotonic Increasingness Property of AMR

AMR is an algorithm that aims at finding meaningful regional minima by merging or filtering useless regional minima. AMR includes two parameters $s$ and $m$. When we increase the value of $m$, gradient images reconstructed by AMR keep the increasing order as shown in Theorem 1.

Theorem 1. Let $\psi$ be an adaptive morphological reconstruction operator, $\psi$ is increasing with respect to the scale of structuring elements, i.e., for a gradient image $g$ such that $f=\varepsilon_{b_{i}}(g)$ and $f\leq g$, $1\leq p$, $q\leq m$, $p,q,m\in N^{+}$, we have

[TABLE]

The proof of Theorem 1 is presented in Appendix B. Theorem 1 shows that the gradient image processed by AMR is monotonous increasing with the increase of $m$. Fig. 9 demonstrates Theorem 1. We can see that if $m$ is enlarged, the more unimportant seeds are removed, and important seeds are preserved. Actually, the result is equivalent to region merging. However, the method is simpler than region merging. According to the result, it can be seen that AMR can help seeded segmentation algorithms to achieve a hierarchical segmentation [47], [48]. Hierarchical segmentation is a multilevel segmentation scheme, and it usually outputs a coarse-to-fine hierarchy of segments ordered by the level of details. Multiscale combinatorial grouping (MCG) proposed by Pont-Tuset et al. [49] is an excellent hierarchical segmentation approach that employs a fast normalized cut algorithm and an efficient algorithm for combinatorial merging of hierarchical regions. Based on the hierarchical segmentation results provided by MCG, some improved approaches are also proposed [50], [51]. These improved approaches achieve better segmentation effect but have lower computational efficiency than MCG.

Before analyzing the relationship between AMR-WT and hierarchical segmentation, we first review some basic concepts of hierarchical segmentation. Let $\Omega$ be a finite set. A hierarchy $H$ on $\Omega$ is a set of parts of $\Omega$ such that

• •

$\Omega\in H$.

• •

For every $\omega\in\left\{\Omega\right\}$, $\left\{\omega\right\}\in H$.

• •

For each pair $(h,h^{\prime})\in H^{2}$, $h\bigcap h^{\prime}\neq\varnothing\Rightarrow h\subseteq h^{\prime}$ or $h^{\prime}\subseteq h$.

Note that $H$ is a chain of nested partitions. Let $H_{0}$ be the initial partition of $\Omega$, which corresponds to the finest partition of $\Omega$, and $H_{n}$ be the coarsest partition of $\Omega$, which segments the images as one single region. A partition $H_{z}$, $0\leq z\leq n$, on $\Omega$ has the property that

[TABLE]

[TABLE]

[TABLE]

where $H_{p}\subseteq H_{q}$ denotes the partition. $H_{p}$ is finer than the partition $H_{q}$. Derived from Theorem 1 and Fig. 9, we obtain

[TABLE]

where $S$ denotes seeded segmentation algorithms such as WT or PW. Suppose that $H_{0}=S(g)$, $H_{m}=S(\psi(g,s,m))$, and $s=1$ then

[TABLE]

According to (12), the principle of the hierarchical segmentation based on AMR is shown in Fig. 11, in which the data points represent regions obtained by the hierarchical segmentation at different levels.

III-C The Convergence Property of AMR

By comparing Fig. 6 with Fig. 8, it can be observed that AMR provides significant gradient images and AMR-WT generates convergent segmentation results via enlarging the scale of structuring elements. An important convergence property of AMR is described in the following.

Theorem 2. Let $\psi$ be an adaptive morphological reconstruction operator, $\psi$ is convergent when increasing the scale parameter $m$, i.e., for any gradient images $f$ and $g$ such that $b_{s}\subseteq\cdots b_{i}\subseteq b_{i+1}\cdots\subseteq b_{m}$, if $min(\psi(g,s,m))\geq max(R_{g}^{\phi}(f)_{b_{m+1}})$ then

[TABLE]

i.e., $\vee_{s\leq i\leq m}\left\{R_{g}^{\phi}(f)_{b_{i}}\right\}=\vee_{s\leq i\leq m+j}\left\{R_{g}^{\phi}(f)_{b_{i}}\right\},1\leq s\leq m,j\in N^{+}$, and the proof is presented in Appendix C.

According to Fig 9, it can be seen that the gradient result and the corresponding seed image will remain unchanged when $m\geq 12$. This empirically illustrates that the gradient image reconstructed by AMR is convergent when increasing the value of $m$. Besides, the large gradient magnitude is unchanged while the small gradient magnitude converges to ones larger than itself for AMR. However, the large gradient magnitude converges to one smaller than itself while the small gradient magnitude converges to one larger than itself for MR when the structuring element is small. With the increase of the value of $m$, the value of gradient magnitudes finally converges to the minimum of the original gradient image, i.e., $\lim\limits_{m\rightarrow\infty}R_{g}^{\phi}(f)_{b_{m}}=min(g)$ (see Appendix A). Consequently, MR removes all regional minima while AMR only filters useless regional minima and preserves significant ones when $m\rightarrow\infty$.

Furthermore, we analyze how to determine the parameter $m$ for AMR. The computational efficiency of AMR is influenced by the parameter $m$. A small $m$ means a low computational complexity. According to Theorem 2, the reconstructed gradient image and the corresponding segmentation result are unchanged when $min(\psi(g,s,m))\geq max(R_{g}^{\phi}(f)_{b_{m+1}})$, but the obtained $m$ is usually large. As the paper aims at employing AMR to improve seeded segmentation algorithms, we replace the convergence condition $min(\psi(g,s,m))\geq max(R_{g}^{\phi}(f)_{b_{m+1}})$ with checking the difference between $\psi(g,s,m)$ and $\psi(g,s,m-1)$. We propose an objective function for justifying the convergence of AMR

[TABLE]

where $m\geq 2$, $m\in N^{+}$. It is clear that the segmentation result will remain unchanged when $J\leq\eta$, $\eta$ is a minimal threshold error, and it is a constant used for $J$, but $m$ is a variant for $\psi(g,s,m)$. Consequently, only a parameter $s$ needs to be tuned for obtaining different reconstruction results.

III-D The Algorithm of AMR

AMR only involves the parameter $s$ and $\eta$, as described in the detailed steps of AMR in Algorithm 1111Source code is available at https://github.com/SUST-reynole/AMR. To speed up the convergence of Algorithm 1, the three parameters $s$, $m$, and $\eta$ are used for AMR because the iteration can be stopped according to $m$ or $\eta$. The computational complexity of AMR depends on the values of $m$ or $\eta$. A large value of $m$ corresponds to a small value of $\eta$. The larger is the value of $m$, the longer is the execution time of AMR. Since we have known that AMR has a fast convergent property as shown in Fig. 8, a small $m$ is enough for moderate-sized images in practical applications. A small m indicates that AMR has a low computational complexity.

Note that the parameter $m$ is unnecessary theoretically, we use two convergent condition $m$ and $\eta$ to speed up the convergence of Algorithm 1. We applied Algorithm 1 to images with complex texture content to demonstrate that the proposed AMR is effective for reducing over-segmentation as shown in Fig. 12. AMR-WT not only overcomes the problem of over-segmentation but also obtains better contours than MR-WT and state-of-the-art superpixel methods. Furthermore, we test Algorithm 1 on images with text to show the monotonic increasing and convergent properties of AMR. Fig. 13 shows the comparison results. We can see that the segmentation results are nested, which demonstrates the monotonic increasing property of AMR. Moreover, the segmentation results are unchanged when $m\geq 11$, which demonstrates the convergent property of AMR.

IV Experiments

To demonstrate the effectiveness and efficiency of the proposed AMR, we apply AMR to seeded image segmentation and spectral segmentation. We conduct experiments on the BSDS500 dataset. The experiments are performed on a workstation with an Intel Core (TM) i7-6700, 3.4GHz CPU and 16GB memory.

We compare the proposed algorithms with state-of-the-art algorithms including a multiscale morphological gradient for watersheds (MMG-WT) [26], multiscale ncut (MNCut) [52], oriented-watershed transform-ultrametric contour map (gPb-owt-ucm) [15], the algorithm recovering occlusion boundaries from an image proposed by Hoiem (gPb-Hoiem) [53], spectral segmentation algorithms proposed by Kim et al. (FNCut, cPb-owt-ucm) [39], Higher-order correlation clustering (HO-CC) [54], global/regional affinity graph (GL-graph) [55], single-scale combinatorial grouping (SCG) [49], and multiscale combinatorial grouping (MCG) [49]. The open source codes and model parameters suggested by the corresponding authors are used. Because the author did not present specific parameter values for MMGR-WT, we set $r=5$ and $0.1\leq h\leq 0.3$, where $h$ is a threshold and it is used to generate a marker image, and $r$ is the radius of the structuring element used for MR. For the proposed approaches, we set $1\leq s\leq 3$, $m=50$, and $\eta=10^{-4}$.

We report the experimental results using three evaluation metrics to quantitatively measure the performance of segmentation algorithms: probabilistic rand index (PRI), segmentation covering (CV), and variation of information (VI). The PRI and CV are similarity measures, and they are large while the VI is small when the final segmentation is close to ground truth segmentation.

IV-A Seeded Image Segmentation

AMR is useful for improving seeded image segmentation because it employs multiscale structuring elements to obtain a convergent seed image without pre-setting many parameters. To show the capability of AMR, it is applied to different gradient images to filter seeds. Fig. 14 shows reconstructed gradient images by AMR and the corresponding segmentation results by WT/PW. These results are clearly better than the ones shown in Fig. 5. The problem of over-segmentation for seeded segmentation algorithms is therefore addressed. Furthermore, compared Fig. 6 to Fig. 14, although both MR and AMR are able to filter seeds, AMR is able to maintain meaningful seeds that correspond to important contours.

Furthermore, Table I shows the number of seeds generated by gradient images. We can see that the reconstructed gradient images generate fewer seeds than original gradient images, which demonstrates AMR is efficient for the filtering of useless seeds. Moreover, AMR is robust for different gradient images obtained by Sobel, gPb, OEF, and SE because the final segmentation results are similar.

In Fig. 14, we set $s=2$ because the segmentation result includes too many small regions when $s=1$. Clearly, $s$ controls the number of small regions in segmentation results. Generally, the value of $s$ depends on the resolution of the images to be segmented, e.g., $1\leq s\leq 3$ for BSDS500.

To demonstrate that the proposed AMR is robust for different images, we implement AMR-WT/PW on the BSDS500. Fig. 15 shows the comparison of segmentation results using different algorithms, i.e., Sobel-AMR-WT/PW, gPb-AMR-WT/PW, OEF-AMR-WT/PW, and SE-AMR-WT/PW. The segmentation results demonstrate the effectiveness of AMR for the filtering of useless seeds, Moreover, AMR is effective for both WT and PW.

To compare the performance of different algorithms on the BSDS500, Table II shows experimental results of three evaluation metrics: PRI, CV, and VI. We can see that AMR is more efficient for improving segmentation results obtained by WT or PW compared to MR. MR is sensitive to $r$ while AMR is insensitive to $s$. Although MMG-MR-WT/PW is effective for the over-segmentation reduction by introducing the parameter $h$, segmentation results are sensitive to both $r$ and $h$. The gPb-AMR-WT/PW, OEF-AMR-WT/PW, and SE-AMR-WT/PW obtain better performance than Soble-AMR-WT/PW since the former provides better gradient images. The SE-AMR-WT/PW obtains the best performance. In addition, AMR-WT obtains higher PRI, CV, and lower VI than AMR-PW in the same situation.

Because AMR converges quickly, AMR has a high computation efficiency for gradient reconstruction. Table III shows the comparison of running time of AMR-WT on different gradient images obtained by Sobel, gPb, OEF, and SE, respectively. We only present the running time of AMR-WT here because AMR-PW has a similar running time as AMR-WT. It can be seen from Table III that AMR-WT has a short running time to achieve image segmentation on the BSDS500. The SE-AMR-WT requires the shortest running time because the corresponding gradient image converges quicker under AMR. Tables II-III show AMR is effective and efficient for improving seeded segmentation algorithms such as WT and PW.

Additional evidence of the superiority of AMR can be found in Fig. 16 which shows experimental results on images with rich texture and faded boundaries. According to Figs. 15-16, we can see that the proposed AMR is effective for different kinds of images.

IV-B Seed-based Spectral Segmentation

In this section, we directly construct the affinity matrix on a pre-segmentation image provided by AMR-WT to reduce the size of the affinity matrix, and then compute the subsequent steps of spectral segmentation (we name it AMR-SC). Note that we employ AMR-WT rather than AMR-PW because the former is able to provide better pre-segmentation results than the latter as shown in Table II. As the pre-segmentation image only consists of dozens of regions, we consider color feature in CIELAB color space and Gaussian function as the criterion to measure the similarity of two regions. Throughout the paper, we use $\sigma=1$. It is clear that the affinity matrix produced by AMR is a small matrix. Therefore, the clusters can be detected easily and fast with the $k$-means algorithm.

In this paper, the pre-segmentation depends on AMR. According to Table II, we set $s=2$ and $3$, and we set the number of clusters for the $k$-means according to [39], [55]. The proposed AMR-SC is evaluated on BSDS500 and compared with algorithms such as gPb-owt-ucm, FNCut, GL-graph, SCG and MCG. Figs. 17-18 show that the proposed AMR-SC generates better segmentation results than those comparative algorithms. The result demonstrates that AMR is useful for improving spectral segmentation due to two reasons. The first is that the regional spatial information of an image provided by pre-segmentation is integrated into spectral segmentation, and the second is that the affinity graph is reduced efficiently by removing useless seeds.

Furthermore, we employ the three measures: PRI, CV and VI to compare the proposed AMR-SC with nine state-of-the-art image segmentation algorithms. Table VI shows the region benchmarks on the BSDS500. In Table VI, the proposed AMR-SC clearly dominates other algorithms on PRI and CV, and is on par with SCG on VI mainly due to accurate pre-segmentation provided by AMR-WT. The OEF-AMR-SC and SE-AMR-SC provide better performance than gPb-AMR-SC and Sobel-AMR-SC because OEF and SE obtain better gradient images than gPb and Sobel. In addition, AMR-SC is insensitive to the parameter $s$.

We tested the running time complexity on the BSDS500 dataset. The running time comparison is shown in Table IV. On average, generating a pre-segmentation result with SE-AMR-WT takes 0.54 seconds (SE generates a gradient image requiring 0.06 seconds. AMR-WT takes 0.48 seconds, $s=3$ and $\eta=10^{-4}$), and constructing an affinity graph and spectral clustering take 0.059 seconds. Consequently, SE-AMR-SC takes about 0.60 second to segment an image from the BSDS500. In contrast, the gPb-owt-ucm takes almost 106.38 seconds, FNCut takes about 10.58 seconds. As GL-graph has four steps, i.e., over-segmentation, feature extraction, bipartite graph construction and graph partition using spectral clustering, it is more complex than SE-AMR-SC, and takes almost 7.41 seconds. MCG takes about 18.60s per image to compute the multiscale hierarchy but SCG takes only 2.21s per image. It is clear that our SE-AMR-SC is the fastest because AMR-SC only depends on the gradient information, and the generated affinity matrix is small.

IV-C Discussion

AMR has two parameters, $\eta$ and $s$. $\eta$ relates to the convergent condition. Generally, a large value of $\eta$ means a few iterations (a small $m$, where $m$ is the number of iterations) while a small value of $\eta$ corresponds to many iterations (a large $m$). Table V shows the influence of $\eta$ on $m$ for test images. We can see that $m$ increases with the decrease of $\eta$ but $m$ is unchanged when $\eta\leq 10^{-4}$.

Furthermore, to show the influence of $\eta$ on AMR, we implement AMR on the BSDS500 by setting different values of $\eta$, and Tables VI-VII show the results. It is clear that the number of iterations for AMR-WT is smaller and running time is shorter if the value of $\eta$ is larger. However, the number of iterations and running time are unchanged when $\eta\leq 10^{-4}$. Therefore, in practical application, users can select different values of $\eta$ according to their requirements.

Furthermore, we implemented SE-AMR-WT on BSDS500 with different values of $\eta$. The performance indices of segmentations are shown in Table VIII. By comparing Tables V-VIII, we can see that the average number of iterations, running time, and segmentation accuracy are unchanged for AMR-WT when $\eta\leq 10^{-4}$. Therefore, the proposed AMR is insensitive to $\eta$.

The value of $s$ controls the initial gradient value of images. A large $s$ will cause the contour offset while a small value of $s$ will cause too many unexpected small regions. Therefore, we choose $s=2$ and $s=3$ for the BSDS500 in Table IV. To further show the influence of $s$ on AMR, Table IX shows the performance indices of segmentations on BSDS500 by setting different values of $s$. It can be seen from Table IX that SE-AMR-WT is insensitive to $s$ if $1\leq s\leq 6$.

V Conclusion

In this work, we have studied the advantages and disadvantages of MR on seeded segmentation algorithms. We proposed an efficient AMR algorithm that can preferably improve seeded segmentation algorithms. The proposed AMR has two significant properties, the monotonic increasingness and the convergence. The monotonic increasingness helps AMR to achieve a hierarchical segmentation. The convergence is able to alleviate the drawback of MR for the filtering of useless regional minima in a gradient image, and guarantees a convergent result. Moreover, we have explored the applications of AMR and have found that AMR is not only able to improve seeded image segmentation results, but also can obtain better spectral segmentation results than state-of-the-art algorithms. Furthermore, the proposed AMR-SC is computationally efficient because a small affinity matrix is used for spectral clustering. Experimental results clearly demonstrate that the proposed AMR-WT generates satisfactory and convergent segmentation results without hard-tuning parameters, and the AMR-SC outperforms most of the state-of-the-art algorithms for image segmentation, and it performs the best in two metrics: PRI and CV.

The segmentation results generated by AMR-WT or AMR-SC can be directly used in object recognition and scene labeling. However, AMR-WT or AMR-SC cannot obtain semantic segmentation results compared to the popular convolutional neural network (CNN), e.g., fully convolutional network (FCN) [56]. To further improve the contour quality of segmentation results, traditional algorithms such as conditional random field [57], image superpixel [58], and spatial pyramid pooling [59], are used to improve the performance of CNN on image segmentation. AMR can be also used in CNN to improve semantic segmentation results. For our future work, we plan to investigate how to combine AMR and FCN effectively and efficiently.

Appendix A Proof of $\lim\limits_{m\rightarrow\infty}R_{g}^{\gamma}(f)_{b_{m}}=max(g)$

Proof:

Since

[TABLE]

we have,

[TABLE]

and

[TABLE]

[TABLE]

According to $R_{g}^{\gamma}(f)=R_{g}^{\delta}(R_{g}^{\varepsilon}(f))$, $R_{g}^{\varepsilon}(f)=\varepsilon_{g}^{n}(f)$ in (1), we get

[TABLE]

Thus,

[TABLE]

In terms of the duality of morphological operation,

[TABLE]

$\hfill\blacksquare$

Appendix B Proof of theorem 1

[TABLE]

Proof:

Let $s\leq p\leq q\leq m$, from Definition 1, we have

[TABLE]

[TABLE]

Because $p\leq q$,

[TABLE]

i.e.,

[TABLE]

$\hfill\blacksquare$

Appendix C Proof of theorem 2

[TABLE]

Proof:

From Definition 1, we have

[TABLE]

[TABLE]

Since $b_{m}\subseteq b_{m+1}\subseteq\cdots\subseteq b_{m+j}$ and $R_{g}^{\phi}(f)_{b_{\infty}}=min(g)$ from Appendix A, we get

[TABLE]

We have known that $min(\psi(g,s,m))\geq max(R_{g}^{\phi}(f)_{b_{m+1}})$, thus

[TABLE]

i.e.,

[TABLE]

$\hfill\blacksquare$