Robust Shape Regularity Criteria for Superpixel Evaluation

TL;DR

This paper introduces a new shape regularity metric for superpixels that considers convexity, repartition, and smoothness, providing a more robust and relevant evaluation than traditional circularity measures.

Contribution

The authors propose a novel superpixel shape regularity metric that improves evaluation robustness and relevance by addressing limitations of existing circularity-based measures.

Findings

The new metric is robust to scale and noise.

It enables more relevant comparison of superpixel methods.

Traditional circularity measures are inadequate for superpixel evaluation.

Abstract

Regular decompositions are necessary for most superpixel-based object recognition or tracking applications. So far in the literature, the regularity or compactness of a superpixel shape is mainly measured by its circularity. In this work, we first demonstrate that such measure is not adapted for superpixel evaluation, since it does not directly express regularity but circular appearance. Then, we propose a new metric that considers several shape regularity aspects: convexity, balanced repartition, and contour smoothness. Finally, we demonstrate that our measure is robust to scale and noise and enables to more relevantly compare superpixel methods.

Figures

Tables

Table 1. Table 1 : Comparison of the superpixel methods regularity averaged on the BSD images for several superpixel scales K = [ 50 , 1000 ] 𝐾 50 1000 K=[50,1000] .

Method	C	SRC
SLIC [6]	$0.438$ $\pm$ $0.111$	$0.518$ $\pm$ $0.072$
ERGC [15]	$0.367$ $\pm$ $0.040$	$0.456$ $\pm$ $0.015$
WP [7]	$0.483$ $\pm$ $0.076$	$0.559$ $\pm$ $0.043$
LSC [8]	$0.228$ $\pm$ $0.046$	$0.327$ $\pm$ $0.035$
SCALP [11]	$0.515$ $\pm$ $0.115$	$0.586$ $\pm$ $0.073$

Table 2. Table 2 : Comparison of the superpixel methods regularity for minimal UE and maximal BR ( K 𝐾 K = 200 200 200 superpixels), and averaged on several superpixel scales K 𝐾 K . See text for more details.

	UE		BR		$K$={50,1000}
Method	C	SRC	C	SRC	C	SRC
SLIC [6]	0.482	0.574	0.360	0.465	0.438$\pm$0.111	0.518$\pm$0.072
ERGC [15]	0.346	0.447	0.346	0.447	0.367$\pm$0.040	0.456$\pm$0.015
WP [7]	0.410	0.507	0.381	0.479	0.483$\pm$0.076	0.559$\pm$0.043
LSC [8]	0.296	0.391	0.202	0.309	0.228$\pm$0.046	0.327$\pm$0.035
SCALP [11]	0.529	0.611	0.389	0.495	0.515$\pm$0.115	0.586$\pm$0.073
Std. dev.	0.095	0.090	0.077	0.074

Table 3. Table 3 : K={50,600}

Method	C	SRC
SCALP-s	0.865$\pm$0.876	0.865 $\pm$0.876
SCALP-o	0.556	0.865
WP	0.865	0.865
SEEDS	0.865	0.865
ERS	0.865	0.865

Equations

$$\text{C}(S)=4\pi |S|/|P(S){|}^2,$$

$$\text{SO}(S)=|S|/|CH|\le 1.$$

$$\text{Vxy}(S)=\sqrt{\min ({\sigma }_x,{\sigma }_y)/\max ({\sigma }_x,{\sigma }_y)}\le 1,$$

$$\text{CO}(S)=|P(CH)|/|P(S)|\le 1.$$

$$\text{SRC}(\mathcal{S})=\sum _k\frac{|S_k|}{|I|}\text{SO}(S_k)\text{Vxy}(S_k)\text{CO}(S_k),$$

Full text

**Index Terms— ** Superpixels, Compactness, Quality measure

1 Introduction

The decomposition of an image into homogeneous areas, called superpixels, has become a very popular pre-processing step in many image processing and computer vision frameworks. For most superpixel-based applications such as object recognition [1, 2], labeling [3] or tracking [4, 5], the use of regular decompositions [6, 7, 8, 9, 10, 11] where superpixels roughly have the same size and regular shapes is necessary. With such regularity, accurate superpixel features can be computed and relevant information can be extracted from their boundaries, while with irregular decompositions [12, 13, 14, 15, 16], superpixels can have different sizes, noisy boundaries and stretched shapes.

Most of recent methods allow the user to tune a compactness parameter [6, 15, 7, 8, 9, 11] to produce superpixels of variable regularity, that may affect the performances for a given application. The search for optimal results and comparison between methods should thus be performed for several regularity settings [17]. Moreover, most superpixel decomposition methods compute a trade-off between adherence to contours and shape compactness. The regularity is hence an argument to compare decompositions with similar contour adherence performances. Hence, clear regularity definition and measure are necessary to evaluate superpixel methods.

In the superpixel literature, the regularity notion is usually called compactness, and is mainly evaluated by the circularity metric [17]. Although the circularity metric has since been considered in many works [18, 15, 10, 11], and large benchmarks [19, 20], the literature usually refers to the regularity as the ability to produce convex shapes with non noisy boundaries. Moreover, for tracking application, the aim is to find one-to-one superpixel associations across decompositions, so compactness should be high for convex shapes with balanced pixel repartition. In [7], the circularity is also discussed since it does not consider the square as a highly regular shape. Figure 1 compares two decompositions computed with [6] using initial square and hexagonal grids. Hexagons provide much higher circularity, although both shapes should be considered as regular. Since most methods start from a square grid and iteratively refine the superpixel borders, it would make sense to have high regularity for a square decomposition, and to have a measure that is in line with the compactness parameter evolution, that produces squares when set to maximum value.

Contributions. In this work, we first demonstrate that the circularity is not adapted to the superpixel context. We propose a new shape regularity criteria (SRC), that better expresses the regularity notion by considering the following aspects: shape convexity, balanced pixel repartition and contour smoothness. Finally, the relevance and the robustness of SRC are demonstrated with state-of-the-art superpixel methods applied to images of the Berkeley segmentation dataset [21].

2 Shape Regularity Criteria

2.1 Regularity Definition

The circularity C, introduced in [17] for superpixel evaluation, expresses the compactness of a shape $S$ as follows:

[TABLE]

where $P(S)$ is the shape perimeter and $|.|$ denotes the cardinality. This metric considers the compactness as the resolution of the isoperimetrical problem that aims to find the largest shape for a given boundary length. As stated in the introduction, the circularity does not express the shape regularity but only favor circular shapes. We propose a new shape regularity criteria (SRC) composed of three metrics, that each evaluates an aspect of the superpixel regularity.

Solidity. To evaluate the global convexity of a shape, we propose to consider its solidity (SO), i.e., the overlap with its convex hull $CH$. Such convex hull, containing the whole shape $S$, is illustrated in Figure 2 and can be computed using Delaunay triangulation. Perfectly convex shapes such as squares or circles will get the highest solidity:

[TABLE]

Balanced repartition. The overlap with the convex hull is not sufficient to express the global regularity. Convex shapes such as ellipses or lines have the highest SO, but should be considered as perfectly regular only with a balanced pixel repartition. To measure it, we define a variance term $\text{V}_{\text{xy}}$:

[TABLE]

with $\sigma_{x}$ and $\sigma_{y}$, the standard deviations of the pixel positions $x$ and $y$ within $S$. \text{\text{V}_{\text{xy}}}=1 if, and only if, $\sigma_{x}=\sigma_{y}$. In this case, the spatial repartition of the pixels around the barycenter is considered as well balanced.

Contour smoothness. Finally, the regularity of the superpixel borders must be considered. The convexity measure (CO) compares the number of boundary pixels of the shape and the one of its convex hull. Although this measure is generally in line with SO, it is mostly dependent on the border smoothness and penalizes noisy superpixels:

[TABLE]

The proposed shape regularity criteria (SRC) is a combination of all regularity aspects and is defined as follows:

[TABLE]

where $\mathcal{S}=\{S_{k}\}_{k\in\{1,\dots,|\mathcal{S}|\}}$ is composed of $|\mathcal{S}|$ superpixels $S_{k}$, whose sizes are considered to reflect the overall regularity. The combination of SO and CO is related to the ratio between the Cheeger constant of the shape $S$ and its convex hull $CH$. As stated in [22], the Cheeger measure is a pertinent tool to evaluate both convexity and boundary smoothness.

2.2 Circularity vs SRC

Figure 3 compares the different metrics on synthetic shapes split into three groups, and generated with smooth (top) and noisy borders (bottom). The circularity presents several drawbacks. First, it is much lower for the Square than for the Circle and Hexagon, and even less than for the Ellipse. As shown in Figure 1, this is an issue when comparing superpixel methods starting from square and hexagonal grids, such as [7]. It also drops for the Cross and Bean although they are visually regular. In the bottom part of Figure 3, the circularity appears to be very dependent on the boundary smoothness, since noisy shapes have similar circularity and cannot be differentiated. Finally, standard shapes with smooth borders can have much higher circularity than regular ones. For instance, the noisy Square has a lower circularity than the Bean.

As can be seen in Figure 3, SO, $\text{V}_{\text{xy}}$ and CO independently taken are not sufficient to express the compactness of a shape. The proposed SRC combines all defined regularity properties. For instance, SO is representative for all shapes, except for the Ellipse and W, since they both have large overlap with their convex hull. $\text{V}_{\text{xy}}$ penalizes the Ellipse since it does not have a balanced pixel repartition, and CO considers the large amount of contour pixels in the W shape.

The three regular shapes get the highest SRC ($\approx 1$), and the standard shapes have similar measures. Since our metric is less sensitive to slight contour smoothness variations, SRC also clearly separates the three shape groups in the noisy case, contrary to C. Moreover, regular but noisy shapes, have comparable SRC to smooth standard ones, whereas noisy standard shapes still have higher SRC than irregular shapes with smooth contours, which can be considered as a relevant evaluation of regularity. Figure 4 also represents the regularity measures, where SRC appears to more clearly separates the three shape groups in both smooth and noisy cases.

Finally, circularity appears to very dependent on the shape size in Figure 5. As observed in [23], due to discrete computation, it can be superior to 1 (we threshold its value in Figure 5), and it drops with larger shapes. Hence, comparisons of methods on this metric would be relevant only with decompositions having the same superpixel number, i.e., superpixels with approximately the same size. Contrary to circularity, SRC provides much more consistent measure according to the superpixel size, e.g., the Square always has a SRC equal to 1.

{comment}

Finally, a decomposition into superpixels can be seen as a preprocessing, similar to the multi-resolution approach, but where the regions are no longer regular blocks. Since most decomposition methods starts from a square grid, and iteratively refine the superpixel borders, it would thus make sense to obtain the highest regularity measure with a square decomposition, i.e., regularity parameter set to infinite.

3 Improved Evaluation of Superpixel Methods

3.1 Validation Framework

To compare results of state-of-the-art methods, we consider the standard Berkeley segmentation dataset (BSD) [21], containing 200 test images of $321{\times}481$ pixels. At least 5 human segmentations are provided to compute evaluation metrics of contour adherence and respect of image objects.

{comment}

{comment}

Methods. We compare the results of the following state-of-the-art methods that enable to set the regularity of their decompositions: SLIC [6], CRS [conrad2013], ETPS [9] and SCALP [11]. All these methods refine the superpixel borders of an initial grid, using color, spatial distance or contour intensity terms (see papers for method specificities). As in Figure 1, we compare the results of SLIC initialized with a square (SLIC-s) and hexagonal grid (SLIC-o). A decomposition example of each considered method is shown in Figure 6.

{comment}

3.2 Evaluation of Superpixel Methods

{comment}

In this section, we consider state-of-the-art methods that enable to set a compactness parameter: SLIC [6], ERGC [15], Waterpixels (WP) [7], LSC [8] and SCALP [11]. Decomposition examples with the associated Delaunay graph are illustrated in Figure 6. In Table 2, we demonstrate that SRC provides a more robust regularity measure of superpixel methods. We compute decompositions at several scales $K$ (from 50 to 1000 superpixels), with the default compactness settings, and average the results on the 200 BSD images. SRC is more robust to the superpixel scale, since it reports lower variance.

We also consider the standard undersegmentation error (UE) metric measuring the number of pixels that belong to several image objects, and the boundary recall (BR), that measures the detection of ground truth contours (defined for instance in [14]). The scale $K$ is set to 300 superpixels, and the UE and BR results are averaged on the BSD images, decomposed with different compactness settings. Hence, for each method, UE and BR results are computed on several regularity levels. We consider the optimal UE and BR of each method, and the standard deviation between the corresponding C and SRC measures are respectively $0.0871$ and $0.0803$ for UE, and $0.0746$ and $0.0694$ for BR. The reduced variance demonstrates the relevance of our metric, since optimal decomposition performances of different methods, evaluated with UE and BR, are obtained for more similar SRC.

{comment}

Image Compression. Since superpixels define a lower-level representation, they can be used to compress the image content [levinshtein2009, zeng2011]. Each superpixel color channel is approximated by a polynomial of n-th order and contained into the polynomial coefficients. We propose to compute these coefficients by a least mean square approach and to evaluate the mean square error between the reconstructed and initial images. A compression example is illustrated in Figure 9.

3.3 Robustness to Noise

In this section, we further demonstrate the robustness of SRC to noisy boundaries. We randomly perturb the trade-off between contour adherence and compactness of SLIC [6] to generate noisy decompositions (see Figure 10). We compute decompositions for different values of compactness parameter $m$, and we report the C and SRC measures averaged on the BSD in Figure 11. The circularity appears to be impacted by the noisy borders, to such an extent that it does not express the global shape regularity that increases with $m$, since it drops from $m=75$. Nevertheless, SRC is proportional to the evolution of the compactness parameter of [6], demonstrating that it better expresses the regularity of a decomposition, and is not mainly sensitive to contour smoothness.

3.4 Global Regularity Evaluation

This work and [17], that introduced circularity, focus on a local compactness definition, where each superpixel is independently evaluated. Such local evaluation seems in line with the visual regularity of the graphs (see Figure 6), but it does not consider the global size regularity within the whole decomposition. Although most methods such as [6, 7, 11] produce superpixels approximately containing the same number of pixels, other methods may produce partitions of irregular sizes [12, 14, 16]. In Figure 12, we represent an example of SLIC superpixels [6] and standard quadtree partition, which produces larger squares in areas with lower color variance. Decompositions are shown with their associated Delaunay graph, connecting barycenters of adjacent superpixels. Since it only produces square areas, the local regularity of such quadtree partition is high ($\text{SRC}=1$), although the associated graph shows inconsistent distances between the barycenters of connected superpixels, contrary to the one of [6]. Numerical evaluation of graph regularity is a complex issue [24], and future works will investigate the global regularity notion and measure in the superpixel context.

4 Conclusion

In this paper, we focus on the notion of regularity, i.e., compactness in the superpixel context. We consider that a regular shape should verify these aspects: convexity, balanced repartition and contour smoothness, and we define a new metric that better expresses the local regularity, and is robust to scale and noise. Most of decomposition methods tend to achieve a trade-off between segmentation accuracy and shape regularity. This work enables to relevantly compare superpixel algorithms and provides accurate regularity information on the decomposition inputs of superpixel-based pipelines. Nevertheless, local compactness measure does not express the global regularity notion, which will be investigated in future works.

{comment}

Finally, we further demonstrate the SRC relevance by showing its correlation with the performances of a superpixel-based labeling framework. In [3], a structure to compare superpixel neighborhoods is defined and applied to exemplar-based labeling. The method uses the Labeled Faces in the Wild (LFW) dataset [huang2007], which contains 927 test images to label and 1500 training images with superpixel labels (face, hair, background). For such application, the regularity of the adjacency graph increases the comparison accuracy between superpixels, the matching relevance, and therefore, the labeling performances [3].

The images are decomposed using SLIC [6] to produce smooth and noisy superpixels with noisy borders (see Figure 13). We compute the labeling performances for several regularity settings and report these results in Figure 14, with the corresponding average regularity. measured by circularity C and by our shape regularity criteria SRC. The circularity is greatly impacted by the noisy borders although it does not alter the global regularity of the superpixel shapes or the adjacency graph complexity. Nevertheless, in both cases, our metric is in line with the labeling performances, demonstrating that it better expresses the regularity of a decomposition.

{comment}