Fuzzy SLIC: Fuzzy Simple Linear Iterative Clustering

TL;DR

Fuzzy SLIC introduces a noise-robust superpixel segmentation method using fuzzy clustering and a novel superpixel number control algorithm, outperforming existing techniques on standard benchmarks.

Contribution

The paper presents Fuzzy SLIC, a superpixel method combining fuzzy clustering with a new control algorithm for precise superpixel count, enhancing noise robustness and efficiency.

Findings

Outperforms state-of-the-art methods on BSD500 and Pascal VOC2007.

Robust against various noise types including Gaussian and salt-and-pepper.

Accurately controls superpixel number without significant computational cost.

Abstract

Most superpixel methods are sensitive to noise and cannot control the superpixel number precisely. To solve these problems, in this paper, we propose a robust superpixel method called fuzzy simple linear iterative clustering (Fuzzy SLIC), which adopts a local spatial fuzzy C-means clustering and dynamic fuzzy superpixels. We develop a fast and precise superpixel number control algorithm called onion peeling (OP) algorithm. Fuzzy SLIC is insensitive to most types of noise, including Gaussian, salt and pepper, and multiplicative noise. The OP algorithm can control the superpixel number accurately without reducing much computational efficiency. In the validation experiments, we tested the Fuzzy SLIC and OP algorithm and compared them with state-of-the-art methods on the BSD500 and Pascal VOC2007 benchmarks. The experiment results show that our methods outperform state-of-the-art techniques in both noise-free and noisy environments.

Figures

Equations

$$\min J(\mathbf{U},\mathbf{c})=\sum _{j=1}^z\sum _{i=1}^{N_j}{u}_{ij}^t\Vert x_i-c_j{\Vert }^2,$$

$$\mathbf{U}\in M_{fc},$$

$$\begin{array}{rl}{M}_{fc}=\{& \mathbf{U}\in V_{z{N}_j}|u_{ij}\in [0,1]\forall i,j;\\ & \sum _{j=1}^z{u}_{ij}=1\forall i;\\ & 0<\sum _{i=1}^{N_j}{u}_{ij}<N_j\forall j\},\end{array}$$

$$F(\lambda ,u_{ij})=J(\mathbf{U},\mathbf{c})+\lambda (\sum _{j=1}^z{u}_{ij}-1).$$

$$u_{ij}={\left(\sum _{k=1}^{m_i}{\left(\frac{\Vert x_i-c_j\Vert }{\Vert x_i-c_k\Vert }\right)}^{\frac{2}{t-1}}\right)}^{-1},$$

$$h_{ij}=\sum _{k\in {\mathcal{W}}_i}{u}_{kj},$$

$$u_{ij}^{\prime }=\frac{u_{ij}^p{h}_{ij}^q}{{\sum }_{k=1}^{m_i}{u}_{ik}^p{h}_{ik}^q},$$

$$c_j=\frac{{\sum }_{i=1}^{N_j}{u^{\prime }}_{ij}^t{x}_i}{{\sum }_{i=1}^{N_j}{u^{\prime }}_{ij}^t}.$$

$$c_j=\frac{{\sum }_{i=1}^{{\mathcal{N}}_j}{u^{\prime }}_{ij}^t{x}_i}{{\sum }_{i=1}^{{\mathcal{N}}_j}{u^{\prime }}_{ij}^t},$$

$$\mathcal{M}=\frac{m^2}{M}+mA,$$

$$d=\Vert c_i-c_j{\Vert }_2,$$

$$BR=\frac{1}{N_g}(\sum _{i=1}^{Ng}\mathrm{logical}(\underset{y_j}{\min }\Vert x_i-y_j\Vert \le 2)),$$

$$\begin{array}{r}CD=\frac{|\mathcal{B}(\mathcal{S})|}{N},\end{array}$$

$$\begin{array}{r}UE=\frac{1}{N}(\sum _{j=1}^z\mathrm{logical}(o_j\mid (o_j\bigcap g_a)\ne \varnothing )\ast \\ \mathrm{logical}(o_j\bigcap g_a\ne \varnothing )\ast |o_j|),\end{array}$$

$$GR(\mathcal{S})=SRC(\mathcal{S})SMF(\mathcal{S}),$$

Full text

I Introduction

The concept of superpixel was first introduced in [1]. Superpixel methods group pixels similar in color and other properties [2]. They capture the redundancy, abstract and preserve the structure from the image [2, 3, 4]. Through substituting thousands of pixels to hundreds of superpixels, they also improve the computational efficiency of subsequent image processing tasks [2, 5, 6]. With the above advantages, superpixel methods are widely used as a preprocessing technique in many image processing tasks like template matching [7], image quality assessment [8], image classification [4, 9], image segmentation [4, 10, 11, 12, 13], etc.

A good superpixel method should have properties, like compactness, partition, connectivity, boundary adherence, low computational complexity, high memory efficiency, controllable superpixel number, and precise generated superpixel number [2]. To achieve these properties, various efficient superpixel methods have been proposed. They can be broadly classified into two categories: graph-based and clustering-based methods [14]. Graph-based methods consider the original image as an undirected graph and use edge-weights obtained as the similarity of color to partition this graph [15]. Normalized cuts (NC) [1] and entropy rate superpixels (ERS) [16] are two typical graph-based superpixel methods. NC adopts cuts while ERS exhibits a bottom-up merging of pixels to generate superpixels [15]. Most of graph-based methods are unable to control the compactness of superpixels and suffered from high computational complexity [15]. Clustering-based methods are often used as a preprocessing method for image segmentation tasks [4, 10, 13]. They consider superpixel generation as a clustering problem. They take color, spatial distance, depth, etc. as the clustering features. Simple linear iterative clustering (SLIC) [2], linear spectral clustering (LSC) [17], simple noniterative clustering (SNIC) [18], superpixels with contour adherence using linear path (SCALP) [19], and superpixel sampling networks (SSN) [14] are typical clustering-based methods. Compared to graph-based methods, clustering-based ones have the advantages of faster speed and controllable compactness. However, most of them need an additional post-processing step to enforce the connectivity [15, 18]. It makes them unable to generate precise number of superpixels as required [18]. Moreover, when applied as preprocessing techniques, superpixel methods often have substantially degraded performance with noisy images [19] as Figs. 1-5 show. Recently, several advanced superpixel methods have been proposed, such as bilateral geodesic distance (BGD) [20], adaptive nonlocal random walks (ANRW) [21], and superpixel optimization using higher order energy (SOHOE) [22], but they are not designed for noisy images and their performance for noisy images still need significant improvement. Although some methods [19, 23] have been proposed to resist noise, they are still not robust enough. Some of them are just designed for specific types of noise only and some others just use a preprocessing step of denoising.

To solve the robustness problem, in this paper, we introduce a local spatial fuzzy C-means clustering method and dynamic fuzzy superpixels, which is called fuzzy simple linear iterative clustering (Fuzzy SLIC). Compared to state-of-the-art methods, Fuzzy SLIC does not need a preprocessing step of denoise and still has a favorable speed. In addition, we introduce a fast and precise superpixel number control algorithm called the onion peeling (OP) algorithm for clustering-based superpixel methods. To validate the proposed methods, we tested Fuzzy SLIC and OP algorithm on Pascal VOC2007 [24] and BSD500 [25] and compared them with ERS, LSC, SCALP, SLIC, and SNIC in five noise conditions: (1) non-noise; (2) salt and pepper noise; (3) Gaussian noise; (4) multiplicative noise; (5) mean filter. The results show that our methods outperform state-of-the-art methods in both noisy and noise-free environment. The contributions of this paper include

This is the first attempt of introducing fuzzy C-means clustering to the clustering-based superpixel method.

• 2.

We have investigated dynamic fuzzy superpixels to efficiently improve the computational performance of the proposed method.

• 3.

We have developed a robust superpixel method which has high performance against different types of noise.

• 4.

A fast and accurate superpixel number control algorithm is proposed for clustering-based superpixel methods.

II Fuzzy Simple Linear Iterative Clustering and Onion Peeling Algorithm

In this section, we will introduce two main parts of the proposed Fuzzy SLIC and a fast and precise superpixel number control algorithm.

II-A Local Spatial Fuzzy C-means Clustering

Fuzzy C-means clustering (FCM) [26] is widely used in the field of image segmentation [13, 27, 28]. However, standard FCM is sensitive to some independent noise points [27]. To overcome this problem, some researchers proposed spatial constrained fuzzy C-means clustering methods (SFCMs) [27, 29]. Through introducing spatial information to FCM, SFCMs can yield homogeneous regions and are more robust against noise than the standard FCM [27]. In this paper, we propose a local spatial fuzzy C-means clustering (LSFCM) which uses a local search instead of a global search. Different to FCM, because of using a local search, the objective function of LSFCM is as follow,

[TABLE]

[TABLE]

[TABLE]

where, $\boldsymbol{U}$ is the fuzzy partition matrix integrated with spatial information which is different to the original FCM, $N_{j}$ is the number of pixels which are in a grid searching region of cluster $j$, $c_{j}$ is the centroid of cluster $j$, $z$ is the number of clusters after initialization which is often different to input cluster number $m$, $t$ is the fuzzy partition matrix exponent which should be larger than 1 for controlling the degree of fuzzy overlap, $u_{ij}$ is the degree of membership of data point $i$ in the cluster $j$, and $\|\cdot\|$ denotes the inner product norm.

Change above optimization problem to an unconstrained optimization problem by introducing Lagrange multipliers as follow,

[TABLE]

By taking partial derivative of $F$ with respect to $u_{ij}$ and $\lambda$ in Eq. (4), we have

[TABLE]

where, $m_{i}$ is the number of possible labels of pixel $i$, because of the local search, only the pixel in a cluster’s searching region is used to update u. Details of the derivation are provided in the supplementary material.

In an image, neighboring pixels are usually highly correlated which means that the probability that these pixels belong to the same cluster is high [27]. Hence, introducing neighboring spatial information can improve the robustness of the clustering methods. LSFCM introduces the spatial information in its fuzzy partition matrix U to compute the new fuzzy partition matrix U’. LSFCM adopts a $3\times 3$ window centered at pixel $i$ to select its neighbors and uses the neighbor’s fuzzy partition degree to construct the spatial function as follow,

[TABLE]

where, $\mathcal{W}_{i}$ is the set of the pixels selected by a square window which centers at pixel $i$. Then combine the H with U to obtain U’,

[TABLE]

where, $p$ and $q$ are the control parameters. A larger $q$ means more important of the effect from neighbors. In this paper, we adopt $p=0$, $q=2$ to achieve high robustness. The selection process of hyperparameters $p$ and $q$ is shown in Section III. Through introducing spatial information to the fuzzy partition matrix, the original membership will be strengthened in a homogenous region. While for a noisy pixel, the weight of a noisy cluster will be reduced by the neighboring spatial information.

Taking the partial derivative of $F$ with respect to $\boldsymbol{c}$ in Eq. (4) and replacing U with U’, then we can update $\boldsymbol{c}$ as follow,

[TABLE]

$\boldsymbol{c}$ is a weighted linear combination of $\boldsymbol{x}$, this is why our algorithm is called fuzzy simple linear iterative clustering. The demonstration of LSFCM is as Fig. 6 shows.

II-B Dynamic Fuzzy Superpixels

How to define the overlapping searching region is a crucial step of LSFCM. We consider two ways: (1) static overlapping region (simple, fast, but less reasonable and not suitable for a clustering method), which is used in a study of improving the performance of TurboPixels [30] and is called fuzzy superpixel (FS) [4]; (2) dynamic overlapping region (reasonable but high computational and memory cost because each pixel has different number of possible labels as Fig. 7 shows and in the clustering process, the maximum number of possible labels of each pixel will also be changed in different iterations). In this paper, we consider dynamic overlapping region but modify it to reduce the computational and memory requirement. We call the dynamic overlapping part of each superpixel’s searching region dynamic fuzzy superpixel (DFS). We find that most pixels do not have more than 3 labels in DFS. We tested different maximum number of possible labels, and we also find when the maximum number of labels of each pixel is set to 3, LSFCM will achieve the best balance of performance and computational cost against FS. The details of the process to determine the maximum number of labels of each pixel in DFS are shown in Section III. With the introduction of DFS, the computational performance of LSFCM will be further improved without performance lost, and the Eq. (8) will be modified as follow,

[TABLE]

where, $\mathcal{N}_{j}$ is the number of pixels which have the label of $j$, this time, it is not a certain value, and different clusters or superpixels $\mathcal{S}$ will have different $\mathcal{N_{S}}$.

The pseudo-code of Fuzzy SLIC is shown in Algorithm 1.

II-C Onion Peeling Algorithm

Clustering-based superpixel methods often need a post-processing step to enforce the connectivity. However, this process leads these methods unable to generate precise number of superpixels and this defect limits the applications of these methods. In many cases, a precise number of superpixels is needed [31]. In this paper, to solve this problem of clustering-based methods, we propose an algorithm called the onion peeling (OP) algorithm. When the number of generated superpixels is not the number as required, OP algorithm will rerun the clustering to generate more superpixels or combine some superpixels to reach the required number. The details of OP algorithm are shown in Fig. 8 and Algorithm 2. In OP algorithm, to generate more superpixels, the algorithm needs to calculate the new cluster number which will be used in superpixel methods as follow,

[TABLE]

where, $M$ is the number of superpixels after enforcing connectivity, and $A$ is an amplification coefficient. In this paper, it is set to 0.2.

The criteria to determine which superpixel will be removed is based on its size. Superpixels will be sorted by their sizes first. Then, superpixels will be removed by the ascending order according to their sizes until required number of superpixels obtained. The removed superpixel will be combined into the most similar one of its directly connecting neighbors. The criteria to determine the similarity between two directly neighboring superpixels is based on the L2-norm, which can be calculated as follow,

[TABLE]

where, $c_{i}$ is the centroid of superpixel $i$ and $c_{j}$ is the centroid of superpixel $j$.

II-D Computational Complexity

Fuzzy SLIC has to update U (fuzzy partition matrix), H (spatial information matrix), and U’ (new fuzzy partition matrix) in its clustering method. The computational time of U and U’ update is proportional to $N$, where $N$ is the number of pixels. The computational time of H update is proportional to $8\times N$ (8 neighbors of each pixel selected by a $3\times 3$ window). The total computational time of Fuzzy SLIC is $O(11N)+O(z)$ which is comparable to SLIC ($O(N)+O(z)$), because $N\gg 10$. The computational complexity of OP algorithm is $O(z-M)+O(N)$, where $z$ is the expected superpixel number. In most cases, the complexity is $O(z-M)\ll O(N)$ since clustering-based methods are usually over-sampling (generated number $\geq$ the required) as Fig. 5 (the last diagram in the second row) shows.

III Experiments and Results

III-A Experiment Settings

To validate the proposed methods, we compared Fuzzy SLIC with ERS, LSC, SCALP, SLIC, and SNIC on BSD500 and Pascal VOC2007 benchmarks, and to ensure the fairness, we set the main parameters of them to the default. For the methods which have the regularity parameter, to make them comparable, we set parameter (Fuzzy SLIC 15, LSC and SCALP 0.3, SLIC and SNIC 20) to achieve similar UE in the noise-free environment. It should be noted that Fuzzy SLIC can achieve more robust results with a higher regularity parameter. A higher regularity parameter can produce a poorer performance in the noise-free environment. To maintain the consistency and fairness, we keep the regularity parameter the same because we also want to achieve the state-of-the-art performance in all situations including noise-free environment when our method is compared with existing ones. The performance of Fuzzy SLIC in the paper can be regarded as a baseline setting of Fuzzy SLIC. To test the robustness of all methods in this paper, we applied them and compared their performance under 5 noise conditions: non-noise, additional Gaussian noise with zero mean standard deviation (std) range $\left[0.1,0.2,0.3\right]$, multiplicative Gaussian noise (mean 0 and std range $\left[0.1,0.2,0.3\right]$), salt and pepper noise with density range $\left[0.1,0.2,0.3\right]$, and a simple mean filter, respectively. To test the OP algorithm, we also integrate it into Fuzzy SLIC and three clustering-based methods: LSC, SCALP, and SLIC. The integrated methods are called Fuzzy SLICNC, LSCNC, SCALPNC, and SLICNC respectively. We also tested the effect of different hyperparameters of Fuzzy SLIC on its performance. Experiments are run on a personal computer with Mac OSX 10.14.1, Intel Core i5 2.3 GHz 4 cores CPU, and 16 GB RAM.

III-B Benchmark Metrics

The evaluation metrics we select are: global regularity (GR) [32], contour density over boundary recall rate (CDBR) [19, 32, 33] and under segmentation error (UE) [30, 34]. We also use the variance of superpixel numbers (VSN) to evaluate the superpixel number control ability of each method. VSN calculate the variance of generated superpixel numbers in a set of same size images using the same $m$.

Boundary recall (BR)

BR is used to evaluate the boundary adherence of superpixel methods. It measures how many boundary pixels of ground truth can be matched by the boundary pixels of a superpixel. If a ground truth boundary pixel is within a distance threshold (usually 2 pixels) of a superpixel boundary pixel, it can be regarded as a hit [1]. The percentage of ground truth boundary pixels matched by superpixel boundaries is the recall rate, and it is ranged from 0 to 1, the higher boundary recall rate means better boundary adherence. Boundary recall rate can be calculated using following formula,

[TABLE]

where, $N_{g}$ is the total number of ground truth boundary pixels, $y_{j}$ is the spatial position of one pixel in superpixel boundary pixel set, and $x_{i}$ is the spatial position of ground truth boundary pixel $i$.

Contour density (CD)

Since BR metric only considers the true positive detection and ignore the density of produced superpixel contours, using this metric only will cause a situation where the shape of superpixel is irregular although it has a high BR value. To overcome this problem, contour density (CD) is proposed to penalize a large number of superpixel boundaries $\mathcal{B(S)}$ [33], the formula of CD is as follow,

[TABLE]

where, $|\mathcal{B(S)}|$ is the number of boundaries of superpixel $\mathcal{S}$, and $N$ is the total number of pixels. We use CD over BR (CDBR) [19, 32, 33] to evaluate the performance of superpixel methods.

Under segmentation error (UE)

UE is another boundary adherence evaluation metric. In an ideal superpixel method, every superpixel should only belong to single object. UE measures the percentage of superpixels which belong to multiple objects. So it is also called leaking rate. It can be calculated as follow,

[TABLE]

where, $o_{j}$ denotes the pixel set of superpixel $j$, $|o_{j}|$ is the number of pixels in superpixel $j$, $z$ is the number of superpixels, $g_{a}$ denotes the pixel set of any ground truth object, and ${\rm logical}(o_{j}\mid(o_{j}\bigcap g_{a})\neq\emptyset)*{\rm logical}(o_{j}\bigcap g_{a}\neq\emptyset)\equiv 1$ means that superpixel $j$ belongs to multiple objects.

Global regularity (GR)

We use the global regularity (GR) [32] to evaluate the shape regularity of superpixels obtained by each superpixel method. The formula of GR is as follow,

[TABLE]

where, $SRC(\mathcal{S})$ is shape regularity criteria of the superpixel $\mathcal{S}$, and $SMF(\mathcal{S})$ is the smooth matching factor of the superpixel $\mathcal{S}$. The detailed formulas of $SRC$ and $SMF$ can refer to [32].

III-C Performance Analysis

Fig. 9††The results for other noise levels of BSD500 and the results of Pascal VOC2007 are included in supplementary material. Our methods achieve similar results of other noise levels in BSD500. Although their performance in Pascal VOC2007 is not as good as in BSD500, their overall performance is still better than other methods. Larger and clearer version of visual and quantitative comparison results are available in supplementary material. shows the CDBR, UE, GR, VSN and average running time curves obtained by all methods used in this paper. We can see that Fuzzy SLIC and Fuzzy SLICNC achieve the best CDBR against state-of-the-art methods in all noise cases. It can be seen that Fuzzy SLIC and Fuzzy SLICNC achieve the second best GR which is just inferior to SCALP and SCALPNC or SLICNC in Gaussian noise, multiplicative noise, and salt and pepper noise. Fuzzy SLIC and Fuzzy SLICNC achieve the best or second best UE in all noise cases. We also can see that all methods integrated with the OP algorithm can control the superpixel number well. In addition, the performance of the methods after integration are similar or a little inferior to the original ones. The average running time of Fuzzy SLIC is better than SCALP which is another robust superpixel method. The running time of the methods after integration except SCALPNC is a little higher than the original ones. Visual comparison results of all methods are shown in Figs. 1-5. We can see that in the cases of noise-free and mean filter, the difference between Fuzzy SLIC, Fuzzy SLICNC, LSC, LSCNC, SCALP, SCALPNC, SLIC, SLICNC, and SNIC is not significant. SCALP and SCALPNC achieve the best performance in Gaussian noise environment while Fuzzy SLIC and Fuzzy SLICNC achieve the second best performance. In the multiplicative noise environment, SCALP, SCALPNC and Fuzzy SLIC achieve the best performance and Fuzzy SLICNC achieve the second best performance. The difference between the first rank and the second rank is not significant. In the salt and pepper noise environment, Fuzzy SLIC and Fuzzy SLICNC achieve the best performance. LSCNC and SLICNC achieve the second best performance. However, the performance of SCALP is not stable in this noise and is much inferior to Fuzzy SLIC. And we can see an interesting phenomenon that some superpixel methods like LSC and SLIC after using OP algorithm are more robust than original ones. It shows that OP algorithm can improve the robustness of some existing superpixel methods to some extent.

III-D Ablation Study of Different Searching Range of Dynamic Fuzzy Superpixels

To select the best searching size of dynamic fuzzy superpixels, we designed an ablation study. We shifted the maximum labels of DFS from 1 to 8 to see the effect of different searching sizes on the performance of Fuzzy SLIC. Fig. 10 shows the comparison results of Fuzzy SLIC with different searching sizes of DFS on the BSD500 without noise. The CDBR of all comparison methods is almost the same, while there is a clear boundary in the curves of GR and UE: That is DFS = 3. We can see that Fuzzy SLIC with DFS = 1 and Fuzzy SLIC with DFS = 2 are significantly inferior to higher DFS searching sizes. There is no obvious difference between the performance of DFS = 3 and higher DFS searching sizes. What’s more, the speed of DFS = 3 is much faster than higher DFS searching sizes. Hence, we use DFS = 3 in our final Fuzzy SLIC in main experiment.

III-E Ablation Study of Different $p$ and $q$ Allocations in Local Spatial Fuzzy C-means Clustering

$p$ and $q$ in the local spatial fuzzy C-means clustering can affect the robustness and the performance of Fuzzy SLIC. To find the best allocation of $p$ and $q$, we designed this ablation study. As paper [27] did, we used 3 allocations of $p$ and $q$: (1) $p=0,q=2$, (2) $p=1,q=1$, (3) $p=2,q=0$ in this study. Figs. 11-14 show the comparison results of different $p$ and $q$ in noise-free and noisy environment. In Fig. 11, we can see that in noise-free environment, all 3 allocations of $p$ and $q$ achieves similar performance on CDBR and UE while Fuzzy SLIC with $p=0,q=2$ achieves the best GR in noise-free environment. Fuzzy SLIC with $p=0,q=2$ outperforms other 2 allocations of $p$ and $q$ significantly in terms of CDBR, UE, and GR in noisy environment as Figs. 12-14 show.

IV Conclusion

This paper proposes a robust superpixel method using local spatial fuzzy C-means clustering and dynamic fuzzy superpixels. The proposed method, compared to state-of-the-art methods, has better performance in terms of boundary adherence, compactness, and computational complexity under a noise-free environment. More importantly, under a noisy environment, the proposed method outperforms state-of-the-art ones significantly. To solve the precise superpixel number control problem of most clustering-based superpixel methods, we propose the OP algorithm. In the validation experiments, the OP algorithm enables a clustering-based superpixel method to control the superpixel number fast and precisely.