Evaluation Framework of Superpixel Methods with a Global Regularity Measure

TL;DR

This paper introduces a comprehensive evaluation framework for superpixel methods, addressing biases in existing metrics by proposing a new global regularity measure and assessing multiple decomposition aspects across various scales and regularity levels.

Contribution

It presents a unified evaluation framework that includes a novel global regularity measure, improving the fairness and robustness of superpixel method comparisons.

Findings

The new global regularity measure (GR) correlates with application performance.

The framework reduces bias in superpixel method evaluation.

Evaluation across multiple scales and regularity levels provides comprehensive insights.

Abstract

In the superpixel literature, the comparison of state-of-the-art methods can be biased by the non-robustness of some metrics to decomposition aspects, such as the superpixel scale. Moreover, most recent decomposition methods allow to set a shape regularity parameter, which can have a substantial impact on the measured performances. In this paper, we introduce an evaluation framework, that aims to unify the comparison process of superpixel methods. We investigate the limitations of existing metrics, and propose to evaluate each of the three core decomposition aspects: color homogeneity, respect of image objects and shape regularity. To measure the regularity aspect, we propose a new global regularity measure (GR), which addresses the non-robustness of state-of-the-art metrics. We evaluate recent superpixel methods with these criteria, at several superpixel scales and regularity levels. The proposed framework reduces the bias in the comparison process of state-of-the-art superpixel methods. Finally, we demonstrate that the proposed GR measure is correlated with the performances of various applications.

Figures

Tables

Table 1. Table 1: Comparison of state-of-the-art superpixel methods, sorted as either graph-based (gb), coarse-to-fine-based (cf) or gradient-ascent-based (ga) approaches. We report the ability of methods to set the number of superpixels, and the regularity of the generated superpixels along with their complexity according to the image size N 𝑁 N .

	TP [5]	ERS [7]	SEEDS [9]	SLIC [8]	ERGC [11]	WP [12]	LSC [13]	ETPS [14]	SCALP [16]
Type	ga	gb	cf	ga	gb	ga	ga	cf	ga
Number	✓	✓	✓	✓	✓	✓	✓	✓	✓
Regularity	✗	✗	✗	✓	✓	✓	✓	✓	✓

Table 2. Table 2: Average EV, ASA and GR on several scales K = [ 25 , 1000 ] 𝐾 25 1000 K=[25,1000] , with the methods default regularity settings.

		TP [5]	ERS [7]	SLIC [8]	SEEDS [9]	ERGC [11]	WP [12]	LSC [13]	ETPS [14]	SCALP [16]
	EV (5)	$0.734$	$0.807$	$0.809$	$0.829$	$0.820$	$0.713$	$0.863$	$0.845$	$0.840$
	ASA (9)	$0.925$	$0.951$	$0.944$	$0.943$	$0.947$	$0.922$	$0.950$	$0.947$	$0.956$
	GR (20)	$0.610$	$0.197$	$0.343$	$0.203$	$0.237$	$0.444$	$0.244$	$0.487$	$0.395$
Optimal results on variable $m$ with $K$=$250$ (Figure 17)	GR	$0.604$	$0.196$	$0.944$	$0.208$	$0.753$	$0.530$	$0.794$	$0.848$	$0.665$
	EV	$0.750$	$0.810$	$0.822$	$0.828$	$0.822$	$0.741$	$0.867$	$0.904$	$0.842$
	ASA	$0.932$	$0.952$	$0.947$	$0.944$	$0.949$	$0.933$	$0.952$	$0.953$	$0.956$
	SPS

Equations

$$D(p,C_k)=d_c(p,C_k)+d_s(p,C_k)\frac{m^2}{r^2},$$

$$D_c(p,C_k)$$

$$\sum _{q\in \mathcal{P}(p)}(F_q-F_{C_k}{)}^2$$

$$=\sum _{q\in \mathcal{P}(p)}{F_q}^2+\sum _{q\in \mathcal{P}(p)}{F_{C_k}}^2-2\sum _{q\in \mathcal{P}(p)}{F}_q{F}_{C_k},$$

$$={{\mathcal{F}}_p}^{(2)}+N\times {F_{C_k}}^2-2\times F_{C_k}\sum _{q\in \mathcal{P}(p)}{F}_q,$$

$$={{\mathcal{F}}_p}^{(2)}+N\times {F_{C_k}}^2-2\times F_{C_k}\times {{\mathcal{F}}_p}^{(1)}.$$

$$\text{ICV}(\mathcal{S})$$

$$\text{EV}(\mathcal{S})$$

$${\text{UE}}_{\text{L}}(\mathcal{S},\mathcal{G})=\frac{1}{|\mathcal{G}|}\sum _{G_j}\frac{\left({\sum }_{S_k,S_k\cap G_j\ne \varnothing }|S_k|\right)-|G_j|}{|G_j|}.$$

$$\text{UE}(\mathcal{S},\mathcal{G})=\frac{1}{|I|}\sum _{S_k}\sum _{G_j}\min \{|S_k\cap G_j|,|S_k\backslash G_j|\}.$$

$$\text{ASA}(\mathcal{S},\mathcal{G})=\frac{1}{|I|}\sum _{S_k}\underset{G_j}{\max }|S_k\cap G_j|.$$

$$\text{BR}(\mathcal{S},\mathcal{G})=\frac{1}{|\mathcal{B}(\mathcal{G})|}\sum _{p\in \mathcal{B}(\mathcal{G})}\delta [\underset{q\in \mathcal{B}(\mathcal{S})}{\min }\Vert p-q\Vert <ϵ],$$

$$\text{ASA}(\mathcal{S},\mathcal{G})=1-\text{UE}(\mathcal{S},\mathcal{G})/2.$$

$$\text{C}(S)=\frac{4\pi |S|}{|P(S){|}^2},$$

$$\text{J}(\mathcal{S})=\frac{1}{|\mathcal{S}|}\sum _{S_k}J(S_k^{\ast },{\hat{S}}^{\ast }),$$

$$S^{\ast }=\frac{1}{|\mathcal{S}|}\sum _{S_k}{S}_k^{\ast }.$$

$$\text{CR}(S)=\frac{\text{CC}(H_s)}{\text{CC}(S)}.$$

$$\text{CO}(S)=|P(H_S)|/|P(S)|\le 1.$$

$$\text{Vxy}(S)=\frac{\min ({\sigma }_x,{\sigma }_y)}{\max ({\sigma }_x,{\sigma }_y)},$$

$$\text{SRC}(\mathcal{S})=\sum _{S_k}\frac{|S_k|}{|I|}\text{CR}(S_k){\text{V}}_{\text{xy}}(S_k).$$

$$\text{SMF}(\mathcal{S})=1-\sum _{S_k}\frac{|S_k|}{|I|}.{\Vert \frac{S^{\ast }}{|S^{\ast }|}-\frac{S_k^{\ast }}{|S_k^{\ast }|}\Vert }_1/2.$$

$$\text{GR}(\mathcal{S})=\text{SRC}(\mathcal{S})\text{SMF}(\mathcal{S}).$$

$$\text{MSE}(I,I_r)=\frac{1}{|I|}\sum _{p\in I}{\left(I(p)-I_r(p)\right)}^2.$$

$$\bar{\mathcal{B}}=\frac{1}{|\{\mathcal{K}\}|}\sum _{K\in \{\mathcal{K}\}}\mathcal{B}({\mathcal{S}}^K).$$

$$\mathcal{C}(p)=\{\begin{array}{ll}\bar{\mathcal{B}}(p)& \text{if}\bar{\mathcal{B}}(p)>\nu \\ 0& \text{otherwise}\end{array},$$

$$\text{F}=\frac{2.\text{P.BR}}{\text{P}+\text{BR}}.$$

$$EV(\mathcal{S})=\frac{{\sum }_{S_k}|S_k|{\left(\mu (S_k)-\mu (I)\right)}^2}{|I|{\sigma }^2(I)}.$$

$$EV(\mathcal{S})=1-\frac{{\sum }_{S_k}|S_k|{\sigma }^2(S_k)}{|I|{\sigma }^2(I)}.$$

$$1-\frac{{\sum }_{S_k}|S_k|{\sigma }^2(S_k)}{|I|{\sigma }^2(I)}$$

$$=\frac{|I|{\sigma }^2(I)-{\sum }_{S_k}|S_k|{\sigma }^2(S_k)}{|I|{\sigma }^2(I)},$$

$$=\frac{X}{|I|{\sigma }^2(I)}.$$

$$X$$

Full text

***Rémi Giraud, \linkableremi.giraud@labri.fr **

1 Introduction

Superpixel decomposition methods, that group pixels into homogeneous regions, have become popular with Ref. 1, and have been widely proposed in the past years [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. Such decompositions provide a low-level image representation, while trying to respect the image contours. The superpixels are usually used as a pre-processing in many computer vision methods such as: object localization [17], stereo and occlusion processing [18], contour detection and segmentation [19], multi-class object segmentation [20, 21, 22, 23, 24], data associations across views [25], face labeling [26], or adapted neural networks architectures [27, 28]. For most methods, the computational cost depends on the number of elements to process, and the superpixel representation is well adapted since this aspect is drastically reduced. Moreover, superpixels can also directly improve the accuracy of applications such as labeling [19], since they gather homogeneous pixels in larger areas, thus reducing the potential noise of independent pixel labeling. Finally, contrary to the classical multi-resolution approach, that decreases the image size by averaging the information, the superpixels preserve the image geometry and content.

The definition of an optimal superpixel decomposition depends on the application. Nevertheless, according to the literature, the desired properties of a superpixel method should be the following. (i) The color clustering must group pixels into homogeneous areas in terms of color features, for instance in RGB or CIELab space. (ii) The decomposition boundaries should adhere to the image contours, i.e., the superpixels should not overlap with several image objects. (iii) The superpixels should have regular shapes and consistent sizes within the decomposition. Regular superpixels provide an easier analysis of the image content, and reach better performances for applications such as tracking [29, 30]. Since these properties cannot be optimal at the same time [31], most decomposition methods compute a trade-off between these aspects in their model. For instance, Figure 1 shows synthetic examples (a) and (d) where a trade-off between the considered aspects must be computed to decompose the images into three superpixels. In Figures 1(b) and (e), the color homogeneity (i) is optimal, while the respect of image contours (ii) and shape regularity (iii) are respectively favored in (c) and (f). Such examples enable to illustrate how a superpixel decomposition method should behave to optimize each criteria.

Many evaluation metrics dedicated to the superpixel context were proposed to evaluate the consistency of a decomposition with respect to the three above-mentioned superpixel properties [5, 7, 8, 32, 33, 34]. The metrics namely include intra-cluster variation (ICV) [35] or explained variation (EV) [4] for color homogeneity (i), the undersegmentation error (UE) [5, 8, 33], achievable segmentation accuracy (ASA) [7], or boundary recall (BR) [36] for the criteria (ii), on the respect of image objects, and circularity (C) [32] or mismatch factor [12] for the criteria (iii), of superpixel shape regularity. These metrics offer different interpretations and evaluations of the superpixel properties. For instance, the circularity [32] and the local regularity metric introduced in Ref. 34 respectively evaluate circular and square shapes as the most regular ones, which can be arguable.

Many recent methods allow the user to set a parameter that enforces or relaxes the regularity constraint of the superpixel shape, e.g., Ref. 8, 11, 12, 13, 14, 16. We show in Figure 2 that a same superpixel method can produce different results, according to this parameter. While regular decompositions provide visually consistent superpixels, with approximately the same shape and size, the irregular one more accurately follows the color variations, at the expense of dissimilar superpixel shapes.

Since the regularity can be set in most recent methods [9, 8, 11, 12, 13, 14, 16], this parameter should be carefully tunned during the evaluation of performances. In each superpixel paper that introduces a decomposition method with such parameter, the authors give a default regularity setting that generally provides a trade-off between segmentation accuracy and superpixel shape regularity. However, when comparing the state-of-the-art methods, even if the results are computed with the default settings of the initial paper, there can be an important bias in the comparison, since most evaluation metrics or superpixel-based pipelines achieve very different performances depending on the regularity of the decompositions.

In the recent study of Ref. 37, no mention is made on the used settings, although the ranking between methods could be altered with different regularity parameters, while Ref. 38 shows that the regularity can be in line with the performances of segmentation and labeling applications. Therefore, we recommend to evaluate the performances with accurate metrics, and for several superpixel scales and regularity levels. Hence, the performances of superpixel methods are not biased by the chosen regularity setting.

1.1 Contributions

In this paper, we propose an evaluation framework of superpixel methods. This work aims to unify the evaluation process across superpixel works and to enable a clear assessment of their performances.

• •

We take a global view of existing superpixel metrics and investigate their limitations. We show that the evaluation can be reduced to one criteria per decomposition property: color homogeneity, respect of image objects and shape regularity.

• •

We address the non-robustness of the state-of-the-art regularity metrics, with the proposed global regularity (GR) measure, that relevantly evaluates the shape regularity and consistency of the superpixels. 111An implementation of the proposed GR metric is available at: www.labri.fr/~rgiraud/downloads

• •

We report an evaluation of the state-of-the-art methods on this GR criteria. Contrary to the standard superpixel literature, to reduce the bias in the evaluation process, we recommend to perform the evaluation for several regularity levels, evaluated with GR, since it expresses the range of potential results of each superpixel method.

• •

Finally, we demonstrate on various applications that the regularity constraint has a substantial impact on performances, and that our GR metric is correlated with the obtained results.

1.2 Outline

In this paper, we first present in Section 2 the existing superpixel metrics, and investigate their limitations. To address the non-robustness of the regularity metrics, we propose in Section 3 a new global regularity measure. In Section 4, we use the considered metrics to compare the state-of-the-art superpixel methods according to several superpixel scales and regularity levels. Finally, we demonstrate the impact of the regularity setting on several applications in Section 5.

Superpixel Decomposition Methods

We first present a non-exhaustive overview of existing superpixel methods. An extensive benchmark evaluation of methods have been proposed in [37]. Nevertheless, this paper mainly focuses on the evaluation of performances on superpixel metrics and does not compare methods according to their regularity parameter. In this work, we show that the regularity of superpixel decomposition is a major criteria to compare methods. We group the decomposition methods as graph, coarse-to-fine, or gradient-ascent-based approaches. Note that the method type does not reflect the properties of the generated superpixels. For instance, as illustrated in Figure 14 two graph-based methods such as Ref. 1 or Ref. 2 respectively produces very regular and irregular superpixels. We report in Table LABEL:table:methods, a summary of the properties of some state-of-the-art methods. We consider the type and computational complexity of each method, and their ability to control the number of computed superpixels and the shape regularity of the superpixels.

1.3 Graph-based Methods

Most of these methods consider the image as a graph and seek a local minimum of a given functional. These approaches enable to produce either regular [1, 11] or irregular decompositions [2]. The term superpixel was introduced in [1] that uses normalized cuts [shi2000] to decompose an image into regular regions. The computational cost of the normalized cuts algorithm, that computes segmentations based on contour and texture features, is substantial and has limited its application.

The methods proposed in [2], performs a fast agglomerative clustering but generates very irregular regions in terms of size and shape. Such irregularity can be an issue, for instance, when trying to compare superpixel neighborhoods (see Section 5.1), since it prevents from matching similar patterns.

In [4], the superpixels are constrained to fit to a lattice, using a contour prior and refines the decomposition using optimal cuts. However, despite this constraint, the produced superpixels have irregular shapes, and the method appears to be very dependent on the input contour map. [6] uses a graph cut approach, and the regularity is enforced by associating pixels to a set of overlapping regular patches at multiple scales. Finally, [7] aims to maximize the entropy rate when cutting the graph. The generated decomposition may be quite irregular, but this method obtains close to state-of-the-art performances.

Recently, [11] proposes an eikonal-based method to iteratively refine the superpixel boundaries. To overcome the initial regular seed placement, a refinement step is considered with a color geodesic distance to add new superpixels.

figure*[ht!]

ETPS [14]

SLIC [8]

Illustration of the steps of the coarse-to-fine [14] (top) and K-means-based iterative clustering [8] (bottom) methods.

Coarse-to-Fine-based Methods

Coarse-to-fine approaches consider the image at different resolution levels and progressively converges towards pixel-level decomposition. A coarse-to-fine framework has been proposed with the SEEDS method [9], that, similarly to the quad-tree based approach, iteratively segments the image by considering smaller block size. SEEDS uses an histogram based energy, which is particularly robust to noise [37, 16], while producing results in reduced computational time. However, this method provides irregular superpixels and does not allow to directly control the number of elements.

More recently, the coarse-to-fine ETPS method were proposed in [14]. Similarly to SEEDS [9], it at each level, the method considers the blocks at the boundaries of superpixel labels, and the transfer is allowed only if it does not violate the connectivity, i.e., cut a superpixel into separate parts. Contrary to [9], ETPS uses enables to set the regularity of the decomposition. In its default settings, it produces regular superpixels, but can generate very irregular ones, as the shape constraint is relaxed. Overall, such coarse-to-fine frameworks enable to achieve low computational burden, by focusing the refinement of the clustering on blocks at the superpixel boundaries.

1.4 Gradient-ascent-based Methods

Methods such as Mean shift [comaniciu2002] and Quick shift [3] use a mode-seeking framework to group pixels into regions. While [comaniciu2002] generates superpixels with highly irregular shapes, in a substantial computational time, [3] considers an initial segmentation to improve the decomposition accuracy, but without explicit control of the superpixel size, and requires more time to perform than [comaniciu2002]. Moreover, these methods present the major drawback of not allowing the control of the superpixel number.

In [5], regular superpixels are generated, with control on their number, using geometric flows starting from a regular grid of initial seeds. Nevertheless, [5] does not allow to set the shape constraint, and produces very regular superpixels at the expense of poor adherence to contours.

Although it does not refer to these computed regions as superpixels, [51] proposed to decompose an image using a watershed approach. With constraints, such watershed methods, that perform a gradient-ascent to a local minima do not constrain the region shapes or sizes. Hence, [51] produces superpixels of highly irregular shapes. To constrain the region shapes, a spatial term is added to the model of [31, 12]. Overall, gradient-based methods enable to produce both very regular [5] and very irregular superpixels [3], but these decompositions do not reach state-of-the-art performances.

1.5 K-means-based Methods

Since its introduction, the simple linear iterative clustering (SLIC) method [8] has been extensively studied. As its name suggests it, the method is straightforward and performs an iterative K-means clustering of pixels starting from an initial grid. The clustering is computed based on a spatial and color distance in the CIELab color space, and can produce either irregular or regular decompositions by setting the trade-off between the two distances. Finally, the SLIC decompositions are computed in order of magnitude faster other approaches [6] or [7] and can perform well on standard superpixel metrics. For these reasons, SLIC has been considered in a substantial number of computer vision pipelines, e.g., medical image segmentation [lucchi2012supervoxel], optical flow computation [lu2013patch], or saliency detection [zhu2014saliency], and has been extended in many works, e.g., Ref. kim2013sp, 31, jia2015, 13, liu2016manifold, agoes2016, 40, huang2016, 15, 16, wang2017, which directly aim to the consistency of the SLIC decompositions with respect to the three above-mentioned superpixel properties.

or superpixel-based object recognition methods such as [20, 23], the use of regular decompositions is mandatory, i.e., the use of methods that provide superpixels with approximately the same size and compact shapes. Regular decompositions provide less superpixel boundaries, producing generally more visually satisfying results. Moreover, most of these methods enable to set the number of superpixels. For superpixel-based video tracking applications [52], regular approaches enable to better handle the time consistency of the decomposition. The tracking of object trajectories within a scene is improved with consistent superpixel decompositions over time [29].

s stated in Section 1, the three superpixel decomposition aspects to evaluate are: homogeneity, regularity and respect of the image objects. In this section, we focus on the SLIC [8] framework, since it is the most used superpixel method and it obtains close to state-of-the-art performances. The recent improvements described in Sections LABEL:sec:slic_homo, LABEL:sec:slic_regu and LABEL:sec:slic_contour aim at optimizing the three decomposition aspects, and enable to reach state-of-the-art performances, as shown in Section 4.

The algorithm of [8] consists of a fast iterative framework that groups pixels into homogeneous regions according to simple color features (average CIELab color). The decomposition starts from a regular grid, composed of blocks of size $r{\times}r$ pixels, with $r=\sqrt{N/K}$, i.e., the ratio between the number of pixels $N$ and the number of desired superpixels $K$. The superpixels $S_{k}$ are each described by an average feature cluster $C_{k}$, which contains the average CIELab colors of pixels $p\in S_{k}$ and the spatial barycenter of $S_{k}$. For each cluster, all pixels within a fixed windows of $(2r+1){\times}(2r+1)$ pixels centered on their barycenter are tested for association at each iteration of the clustering process. The superpixel size is hence constrained to lie within a bounding box. The pixel $p$ is compared to a cluster $C_{k}$ using a spatial distance $d_{s}$, and a color distance $d_{c}$:

[TABLE]

with $m$ the regularity parameter that sets the compromise between the color distance $d_{c}$ and the spatial distance $d_{s}$ in $D$. High values of $m$ produce more regular superpixels, i.e., closer to the initial square grid, while small values of $m$ allow the superpixels to better fit to the image contours at the expense of higher variability in the superpixel sizes and shapes. The pixel $p$ is associated at each iteration to the superpixel $S_{k}$ that minimizes $D$, the sum of the color and spatial distances. We illustrate the iterative clustering process of [8] in the bottom part of Figure 1.3, starting from a regular grid and iteratively converging towards superpixels that fit to the image content. This approach is performed on simple color features and achieves close to state-of-the-art results in a reduced computational time. Many approaches have then been proposed to improved this framework, generally by adding features to (1).

figure[t]

(a) (b) (c)

Illustration of the iterative clustering algorithm [8] after [math] (a), $1$ (b) and $10$ (c) iterations.

To enforce the color homogeneity within the SLIC model, recent works propose to compute content-sensitive superpixels using manifolds which are embedded in the spatial and color space [liu2016manifold], while other approaches improve the performances by computing the clustering in a high dimensional feature space, e.g., [13, 16]. Several works proposed to consider a geodesic color distance along the pixels towards the superpixel barycenter [52, 15]. Such term may further enforce the homogeneity since the colors along the path must also be close to the average superpixel color.

imilarly to patch-based approaches, the pixels in a square area, or patch, centered on the current pixel are considered in the proposed distance $D_{c}$:

[TABLE]

with $\mathcal{P}(p)$ the square patch centered on $p$, and $F$, the image features (e.g., in [8], $F=[l,a,b]$). Note that a Gaussian kernel $G_{\beta}$, defined in (LABEL:filtering), is also used to favor the contribution of the closest pixels.

The complexity of the proposed distance (2) is $\mathcal{O}(N)$, with $N=|\mathcal{P}(p)|$, the number of pixels in the neighborhood.

proof The distance between features $F$ in (2) reads:

[TABLE]

The terms ${\mathcal{F}_{p}}^{(2)}=\sum_{q\in\mathcal{P}(p)}{{F_{q}}^{2}}$, and ${\mathcal{F}_{p}}^{(1)}=\sum_{q\in\mathcal{P}(p)}{F_{q}}$, which only depend on the initial image, can be pre-computed. The complexity of the proposed distance $D_{c}$ is hence reduced to $\mathcal{O}(1)$ instead of $\mathcal{O}(N)$.

Most methods generally consider a regularity parameter to set the superpixel regularity. As can be seen in Figure 2, the SLIC method [8] can produce irregular shapes. In [16], the information of pixels on the linear path [bresenham1965] to the superpixel barycenter is considered to enforce the regularity.

figure[h]

(a) (b) (c)

Example of SLIC decomposition [8] (a) with irregular shape (b), set regular with the color distance on the linear path (LABEL:newdist0) of [16] (c).

ince the optimal color homogeneity may be not correlated with the respect of image objects, a contour information can also be used on such direct path [16, wang2017]. In [wang2017], a binary contour map is used to prevent the association of a pixel to a superpixel, if an edge is crossed on the direct path to the superpixel barycenter. In [16], the maximum of intensity of a smooth contour prior, is considered on the linear path and included in the clustering distance of [8].

2 Standard Superpixel Metrics and Limitations

In this section, we present the existing superpixel metrics, that were progressively introduced in the literature to evaluate the color homogeneity, the respect of image objects or the shape regularity, and we show their limitations to only focus on relevant ones. In Sections 2.1 and 2.3, we present color homogeneity and shape regularity metrics that evaluate for an image $I$, a superpixel decomposition $\mathcal{S}=\{S_{k}\}_{k\in\{1,\dots,|\mathcal{S}|\}}$ composed of $|\mathcal{S}|$ superpixels $S_{k}$, where $|.|$ denotes the cardinality of the considered element. In Section 2.2, we present metrics that evaluate the respect of image objects and compare the decomposition to a ground truth denoted $\mathcal{G}=\{G_{j}\}_{j\in\{1,\dots,|\mathcal{G}|\}}$, with $G_{j}$ a segmented region.

2.1 Homogeneity of Color Clustering

The homogeneity of the color clustering is a core aspect of the superpixel decomposition. Most methods compute a trade-off between spatial and color distances to compute the superpixels. The ability to gather homogeneous pixels should hence be considered in the comparison process, but this aspect is rarely evaluated in state-of-the-art superpixel works. In Section 5, we show that color homogeneity is particularly interesting for image compression [5, 39].

The homogeneity of colors can be evaluated by comparing the average colors of superpixels to the colors of the pixels in the initial image. The intra-cluster variation (ICV) [35] has been proposed to measure such color variation for a decomposition $\mathcal{S}$:

[TABLE]

with $\mu(S_{k})$, the average color of the superpixel $S_{k}$.

The explained variation (EV) [4] was also proposed to evaluate the homogeneity of the color clustering and is defined as:

[TABLE]

Limitations

The ICV metric presents several drawbacks. It is not normalized by the image variance to allow comparable evaluation between different images [37], and the superpixel size is not considered, so the measure is not robust to the superpixel scale. In Figure 4, we illustrate these issues on synthetic examples. The dynamic and size of the same image with the same decomposition are altered, and these transformations impact the measure of ICV.

The EV metric is robust to these transformations. It expresses the color variance contained into each superpixel, and simple calculations provide another explicit formulation:

[TABLE]

In Figure 5(c), we illustrate the color variance $\sigma(S_{k})^{2}$ within each superpixel $S_{k}$. Higher variance within a superpixel will lower the EV criteria, so high EV values express homogeneous color clustering. Despite the robustness of this criteria, it was not considered in the main state-of-the-art superpixel works. As in Ref. 37, we recommend the use of EV to evaluate color homogeneity.

2.2 Respect of Image Objects

The most considered aspect in the superpixel evaluation is the respect of the image objects. An accurate decomposition should have superpixels that do not overlap with multiple objects. To evaluate this aspect, many metrics were proposed. In the literature, undersegmentation error (UE), achievable segmentation accuracy (ASA) and boundary recall (BR) measures are mainly reported. While UE and ASA compute overlaps with the ground truth regions, BR is a contour adherence measure, which is usually correlated to the first two metrics.

Regarding the UE, several definitions were proposed [5, 8, 33]. UE evaluates the number of pixels that cross ground truth region boundaries. The initial UE formulation [5], denoted $\text{UE}_{\text{L}}$, is defined as:

[TABLE]

The $\text{UE}_{\text{L}}$ measure was discussed in several works, e.g., Ref. 8, 9, 33, since any superpixel that has an overlap with the ground truth segment penalizes the metric. Therefore, $\text{UE}_{\text{L}}$ is very sensitive to small overlaps and does not accurately reflect the respect of the image objects. A method to reduce the overlap considered in (7) was proposed in Ref. 8, but it requires a parameter. Recent state-of-the-art works tend to use the free parameter formulation of UE introduced by Ref. 33:

[TABLE]

This formulation respectively considers the intersection or the non-overlapping superpixel area in case of small or large overlap with the ground truth region, and addresses the non-robustness of former UE definitions.

The ASA measure [7] also aims at evaluating the overlap of superpixels with the ground truth. It is reported in most of the superpixel literature as follows:

[TABLE]

For each superpixel $S_{k}$, the largest possible overlap with a ground truth region $G_{j}$ is considered, and higher values of ASA indicate better results.

Another measure is extensively reported in the superpixel literature to evaluate the adherence to object contours: the boundary recall (BR). This metric evaluates the detection of ground truth contours $\mathcal{B(\mathcal{G})}$ by the superpixel boundaries $\mathcal{B}(\mathcal{S})$ such that:

[TABLE]

with $\delta[a]=1$ when $a$ is true and [math] otherwise, and $\epsilon$ is a distance threshold that has to be set, for instance to $2$ pixels [7, 9]. Each ground truth pixel is considered as detected, if a superpixel boundary is at less than an $\epsilon$ distance.

Limitations

Although the BR measure (10) has been extensively reported in the literature and is recommended in Ref. 37, it does not express the respect of image objects or the contour detection performances, as it only considers true positive contour detection. Hence, the number of computed superpixel contours is not considered, and very irregular methods can obtain higher BR results. Figure 6 compares two decompositions with the same number of superpixels that have maximal $\text{BR}=1$. One of the decomposition is very irregular and produces a high number of boundary superpixels, and this aspect is not considered in the BR metric. For these reasons, recent works such as Ref. 12, 40, 16 report BR results according to the contour density (CD), which measures the number of superpixel boundaries such that $\text{CD}(\mathcal{S})=|\mathcal{B}(\mathcal{S})|/|I|$. Nevertheless, BR needs a parameter $\epsilon$ to be set and is not sufficient to reflect the respect of image objects. BR should only be considered as a tool for evaluation of contour detection performances, as shown in Section 5.

The study of Ref. 37 claims that the UE (8) and ASA (9) measures are correlated, so that $\text{ASA}=1-\text{UE}$, making redundant the use of both for superpixel evaluation. Under relevant assumptions, we now show that the true relation between the two measures is:

[TABLE]

In Appendix A, we demonstrate that the relation (11) is exact when all superpixels have a major overlap, i.e., a ground truth region intersects with more than half of the superpixel area. This case is illustrated in Figure 7(b), where a superpixel $S_{k}$ has a major overlap with a region $G_{2}$. Note that this assumption is necessarily true when each superpixel overlaps with only two regions or when the ground truth is binary. In Appendix A, we measure the error of (11) on state-of-the-art methods. This error appears to be negligible, underlying the likelihood of this assumption.

Hence, since the UE${}_{\text{L}}$ measure has been proven to be non-robust, and we demonstrate that the study of UE and ASA is equivalent, we recommend to restrain the evaluation of the respect of image objects to the ASA results (9).

2.3 Regularity

Superpixel decompositions compute a lower-level image representation. Since the desired number of elements is given, and most methods tend to compute regions with approximately the same size, superpixels are generally created in smooth areas (see for instance Figure 2). To produce a consistent decomposition, most superpixel methods compute a trade-off between color clustering and regularity, which has been proven to have an impact on application performances [6, 30, 38]. Therefore, the regularity of the superpixel shapes is a core aspect when evaluating and comparing decomposition methods.

In Ref. 32, the circularity metric was introduced to locally evaluate the compactness of the superpixels. This measure is the usual local regularity metric, and has been considered in state-of-the-art works [30, 11, 40, 16], and benchmarks [41, 37, 34]. This circularity C is defined for a superpixel shape $S$ as follows:

[TABLE]

where $P(S)$ is the superpixel perimeter. The regularity is hence considered as the ability to produce circular areas.

While the circularity independently evaluates the local compactness of each superpixel, other works propose to evaluate the consistency of the shapes within the decomposition. In Ref. 34, the variance of superpixel sizes is considered. Ref. 12 goes further and proposes an adapted version of the mismatch factor [42] to measure the consistency in terms of size and shape. The mismatch factor is computed as $1-J(S_{1},S_{2})$ with $J$ being the standard Jaccard index [43], that computes the overlap between two regions $S_{1}$ and $S_{2}$ such that: $J(S_{1},S_{2})=|S_{1}\cap S_{2}|/|S_{1}\cup S_{2}|$. The global measure J for a superpixel decomposition $\mathcal{S}$ is computed as follows [12]:

[TABLE]

with $S_{k}^{*}$ the registered superpixel $S_{k}$, so its barycenter is at the origin of the coordinates system, and $\hat{S}^{*}$, the binary average shape of $\mathcal{S}$. In the following, we consider the J metric when comparing to Ref. 12, since, contrary to the mismatch factor, its interpretation is consistent with the other presented metrics, i.e., higher value is better. Eq. (13) compares each superpixel to the binary average shape $\hat{S}^{*}$ of the decomposition using $J$. To compute $\hat{S}^{*}$, the registered superpixels $S_{k}^{*}$ are first averaged into $S^{*}$ such that:

[TABLE]

A thresholded shape $S_{t}^{*}$ is then defined by binarization with respect to a threshold $t$. The binary average shape is finally defined as $\hat{S}^{*}=S_{\underset{t}{\text{argmax}}{(|S_{t}^{*}|\geq\mu)}}^{*}$, with $\mu=|I|/|\mathcal{S}|$, the average superpixel size. We illustrate these definitions in Figure 8. We represent a superpixel decomposition $\mathcal{S}$, the corresponding average of superpixel shapes $S^{*}$, and the binary average shape $\hat{S}^{*}$.

Limitations

Although the circularity metric (12), introduced in Ref. 32, is the local regularity metric usually used in the literature [30, 11, 40, 16, 41, 37], it presents several drawbacks. For instance, the relevance of this metric, and the regularity definition have been discussed [12], since the circularity considers a circle or a hexagon as more regular than a square, and is very sensitive to boundary smoothness [44]. As a consequence, some methods, such as Ref. 40, start from a hexagonal repartition of superpixels seeds, and design their spatial constraint to fit to a hexagonal grid in order to obtain higher circularity. However, the superpixel literature more generally refers to the regularity as the ability to produce convex shapes with smooth boundaries. Therefore, squares, circles or hexagons should be considered as regular shapes. Moreover, since most superpixel methods generate a regular square grid with their regularity parameter set to maximum value, it would make sense to obtain the highest regularity measure for such square decomposition.

The mismatch factor or Jaccard index (13) was introduced by Ref. 12 to evaluate the shape consistency within the whole decomposition, but it does not consider the superpixel size, so each shape equally contributes. Moreover, by computing a thresholded average shape, the metric appears to be non-robust to large shape outliers, leading to potential irrelevant measures. Finally, with such metric that only considers the shape consistency, decomposing an image with its lines or with stretched rectangles would give the highest regularity measure.

Contrary to ASA and EV, that give a relevant measure of aspects (i) color homogeneity and (ii) respect of image objects, no existing metric provides a robust and accurate regularity measure of a superpixel decomposition. As a consequence, we propose in the following Section 3, a new global regularity measure, that addresses the limitations of state-of-the-art ones.

3 A new Regularity Measure

In the literature, the circularity (12) has been proposed to measure the shape regularity [32], and the mismatch factor to evaluate the shape consistency across the decomposition [12]. These measures present several drawbacks, that we address in this work by introducing two new metrics (18) and (19), combined into the proposed global regularity (GR) measure. GR robustly evaluates both shape regularity and consistency over the decomposition, and we demonstrate in Sections 4 and 5 that it relevantly evaluates the performances of superpixel methods.

3.1 Shape Regularity

As stated in Section 2.3, an accurate superpixel shape regularity measure should provide the highest results for convex shapes, such as squares, circles or hexagons, and penalize unbalanced shapes while considering noisy boundaries. To express such a measure, we propose to combine all these aspects into a new shape regularity criteria (SRC) [44]. The convexity of a shape $S$ is considered, i.e., the smoothness of its contours and the overlap with its convex hull $H_{S}$, that entirely contains $S$. To evaluate both overlap with the convex hull and contour smoothness, we first define as $\text{CC}(S)=|P(S)|/|S|$, the ratio between the perimeter and the area of a shape $S$, which is linked with the Cheeger constant for convex shapes [45]. Then, we introduce our criteria of regularity (CR) as:

[TABLE]

Since the convex hull $H_{s}$ entirely contains $S$ and has a lower perimeter, the CR measure is between 0 and 1, and is maximal for convex shapes such as squares, circles or hexagons. Contour smoothness. Finally, the regularity of the superpixel borders must be considered. The convexity measure (CO) compares the number of boundary pixels of the shape and the one of its convex hull. Although this measure is generally in line with SO, it is mostly dependent on the border smoothness and penalizes noisy superpixels:

[TABLE]

Nevertheless, the comparison to the convex hull is not sufficient to define the regularity of a superpixel.

The balanced repartition of the pixels within the shape is another aspect to consider. Otherwise, shapes such as ellipses or lines would get the maximum CR results. We define the variance term $\text{V}_{\text{xy}}$ as the ratio between the minimum and maximum variance of pixel positions $x$ and $y$ that belong to $S$:

[TABLE]

with $\sigma_{x}$ and $\sigma_{y}$, the standard deviations of the pixel positions. Such measure enables to penalize dissimilarity in the pixel repartition, and \text{\text{V}_{\text{xy}}}=1 if, and only if, $\sigma_{x}=\sigma_{y}$, i.e., if the spatial pixel repartition is balanced.

The proposed shape regularity criteria (SRC) is defined as follows:

[TABLE]

Note that in practice, we use the square root of $\text{V}_{\text{xy}}$, so both criteria have similar variation ranges. SRC robustly and jointly evaluates convexity, contour smoothness and balanced pixel repartition.

figure[t!]

[TABLE]

Convex hull example on a synthetic shape. The overlap between the shape (a) and its convex hull (b) is shown in (c). The shape is contained into the hull and the overlap is such that $\text{SO}=0.78\%$.

Circularity vs Shape Regularity Criteria

To demonstrate the robustness and relevance of SRC (18) over circularity C (12), we consider in Figure 9 synthetic shapes that are split into three groups (regular, standard and irregular), and generated with smooth (top) and noisy boundaries (bottom). First, we present the circularity drawbacks, which reports lower measure for the Square than for the Hexagon, or the Ellipse. Since methods such as Ref. 12, 40 produce superpixels from a hexagonal grid, the regularity evaluation is very likely to be superior for these methods than for other ones starting from square regular grids. The circularity is also very sensitive to the contour smoothness, so regular and standard noisy shapes have similar measure, and the groups are no longer differentiated (see the bottom part of Figure 9). Finally, standard but smooth shapes have much higher circularity than noisy regular ones.

s can be seen in Figure 9, SO, $\text{V}_{\text{xy}}$ and CO independently taken are not sufficient to express the compactness of a shape. The proposed SRC combines all defined regularity properties. For instance, SO is representative for all shapes, except for the Ellipse and W, since they both have large overlap with their convex hull. $\text{V}_{\text{xy}}$ penalizes the Ellipse since it does not have a balanced pixel repartition, and CO considers the large amount of contour pixels in the W shape.

With SRC, we first note that the three regular shapes have the highest regularity measure ($\approx 1$), and that regular but noisy shapes, have similar SRC to the smooth standard ones. Overall, since SRC is less dependent on the boundary smoothness, in each group, smooth and noisy shapes are clearly separated, contrary to C. Finally, we show the metric evolution with the shape size in Figure 10. As stated in Ref. 46, due to discretization issues, the circularity must be thresholded so it is not superior to 1, and it drops as the shape size increases. Therefore, this metric is not robust to the superpixel size, and the comparison of methods on circularity is relevant only if the compared superpixel decompositions have the same number of elements. Contrary to the circularity, the SRC metric is robust to the superpixel scale and provides a consistent evaluation of shape regularity.

figure[t!]

[TABLE]

Comparison of the circularity (C) and the proposed shape regularity criteria (SRC) on the ability to separate the three groups of noisy shapes (a) and the robustness to the superpixel scale (b).

inally, a decomposition into superpixels can be seen as a preprocessing, similar to the multi-resolution approach, but where the regions are no longer regular blocks. Since most decomposition methods starts from a square grid, and iteratively refine the superpixel borders, it would thus make sense to obtain the highest regularity measure with a square decomposition, i.e., regularity parameter set to infinite.

Nevertheless, the SRC and circularity metrics independently evaluate each superpixel shape without considering the global homogeneity of the decomposition. Superpixel methods, e.g., Ref. 2, 15, or other segmentation algorithms such as the quadtree partitioning method [47] can produce regions of very variable sizes. In Figure 11, we show an example of decomposition with superpixels having approximately the same size [8], and a standard quadtree-based partition [47], which produces larger regions in areas with lower color variance. Since the circularity and SRC measures independently consider each superpixel and report an average evaluation of local regularity, the quadtree partition obtains the highest measure although its elements do not have similar sizes. Such local metrics are thus not sufficient to express the global regularity of a decomposition.

3.2 Shape Consistency

The regularity of the superpixel shapes evaluated with a local criteria is a relevant information but does not reflect their consistency in terms of shape and size within the decomposition. In Section 3.1, we define the shape regularity properties for a superpixel: convexity, boundary smoothness and balanced pixel repartition. Nevertheless, a perfectly regular decomposition of an image should be composed of similar regular shapes at the same scale. In other words, a relevant criteria should locally evaluate the shape regularity and how the superpixels are consistent in terms of shape and size within the decomposition.

As stated in Section 2.3, the mismatch factor (13) uses the Jaccard index to compare the superpixels to a thresholded average shape. As illustrated in Figure 12, this measure can be incoherent with the visual consistency of a decomposition. In these examples, the binary average shape $\hat{S}^{*}$ is the same, and corresponds to the red areas. The J measure is low and incoherently decreases as the consistency is visually improved.

In this work, we propose the smooth matching factor (SMF), that directly compares the superpixels to the average shape $S^{*}$:

[TABLE]

SMF compares the spatial distributions of the average superpixel shape $S^{*}$ to each registered superpixel shape $S_{k}^{*}$. The SMF criteria should be close to 1 if the distributions of pixels within the shapes are similar, and close to [math] otherwise. Overall, the proposed SMF metric evaluates the consistency in terms of shape and size within a decomposition. By considering the average shape $S^{*}$ without thresholding, the evaluation is more robust to shape outliers. This criteria addresses the non-robustness of the mismatch factor (J) [12], as can be seen in Figure 12, where SMF is relevant according to the consistency of the decomposition. Finally, examples of superpixel decomposition on natural images are given in Figure 13 and illustrate that SMF is not correlated to the J measure.

3.3 Global Regularity Measure

As previously stated, a perfectly regular superpixel decomposition should be composed of compact shapes, that would be consistent in terms of size and shape. The SRC metric enables to locally evaluate the shape regularity while the SMF measures the shape consistency. To evaluate both aspects, we propose to combine these metrics in the global regularity (GR) measure:

[TABLE]

In the following Section 4, we report the evaluation on the considered metrics (EV, ASA, GR) for state-of-the-art superpixel methods. These measures are reported according to the number of generated superpixels, and also according to the regularity, since this setting substantially impacts performances.

4 Comparison of State-of-the-Art Superpixel Methods

4.1 Dataset

We compare the performances of state-of-the-art methods on the standard Berkeley segmentation dataset (BSD) [48]. This dataset contains 200 various test images of size $321{\times}481$ pixels. For each image, human segmentations are provided and considered as ground truth to evaluate the respect of image objects. At least five decompositions are provided per image, and the presented results in the following sections are averaged on all ground truths. Note that other datasets can be considered, e.g., Ref. 49, 50, but the BSD [48] is the most used dataset for comparing superpixel methods. Moreover, Ref. 37 shows that decomposition algorithms that perform well on the BSD usually perform well on other datasets.

4.2 Considered Superpixel Methods

Many frameworks have been proposed to decompose an image into superpixels using either graph-based [2, 6, 7, 11], watershed [51, 31, 12], coarse-to-fine [9, 14] or gradient-ascent [3, 5, 8, 13, 16] approaches (see Ref. 37 for a detailed review of existing methods). In order to illustrate the proposed evaluation framework, and to validate our regularity measure, we consider the following state-of-the-art superpixel methods: TP [5], ERS [7], SLIC [8], SEEDS [9], ERGC [11], WP [12], LSC [13], ETPS [14] and SCALP [16]. As stated in the introduction, only the most recent methods, SLIC [8], ERGC [11], WP [12], LSC [13], ETPS [14] and SCALP [16], enable to set a regularity parameter. A decomposition example for each considered method is given in Figure 14.

\stab

3mm

4.3 Quantitative Results

figure[t!]

[TABLE]

Comparison of circularity (C) and proposed shape regularity criteria (SRC) on shapes of various pixel sizes $p$.

In this section, we compare the methods considered in Section 4.2 on the recommended metrics for several superpixel scales and several regularity levels. In Figure 16, we perform the standard evaluation of performances, according to the number of generated superpixels. The behavior of each method regarding the different decomposition aspects, homogeneity of color clustering (EV), respect of image objects (ASA), and regularity (GR), is respectively evaluated in Figures 16(a), (b) and (c). Methods such as TP [5] and WP [12], that produce very regular superpixels appear to poorly perform on other metrics. Although they report high regularity, recent methods SCALP [16] and ETPS [14] perform well on color homogeneity, evaluated with EV. SCALP [16] even performs best on respect of image objects, evaluated with ASA. We summarize this evaluation in Table 4.3. A superpixel decomposition cannot obtain maximum values on each criteria at the same time, and according to the desired decomposition aspect, one would grant more consideration to a particular criteria.

inally, as demonstrated in Section 3, we show in Figure 15 that the proposed GR measure differs from the J one [12]. The proposed GR provides much smoother evaluation of regularity according to the superpixel size, especially for Turbopixels (TP) [5] and Waterpixels (WP) [12] methods. Moreover, the hierarchy between methods is modified with our robust evaluation metric. For instance, SEEDS [9] gets similar results to ERS [7] and SCALP [16] gets higher results than SLIC [8]. This shows that both measures are not equivalent, and we demonstrate in Sections 3, 4 and 5 the robustness and relevance of the GR metrics that should be considered for regularity evaluation of superpixel methods.