Saliency detection based on structural dissimilarity induced by image quality assessment model

TL;DR

This paper introduces a novel saliency detection method that measures structural dissimilarity based on image quality assessment principles, outperforming existing models by focusing on structural features like contrast and gradient.

Contribution

The paper proposes a new saliency detection approach using structural dissimilarity derived from IQA models, emphasizing structural features over traditional feature difference methods.

Findings

Outperforms 11 state-of-the-art models on three databases

Uses structural features like contrast and gradient magnitude

No postprocessing needed for high performance

Abstract

The distinctiveness of image regions is widely used as the cue of saliency. Generally, the distinctiveness is computed according to the absolute difference of features. However, according to the image quality assessment (IQA) studies, the human visual system is highly sensitive to structural changes rather than absolute difference. Accordingly, we propose the computation of the structural dissimilarity between image patches as the distinctiveness measure for saliency detection. Similar to IQA models, the structural dissimilarity is computed based on the correlation of the structural features. The global structural dissimilarity of a patch to all the other patches represents saliency of the patch. We adopt two widely used structural features, namely the local contrast and gradient magnitude, into the structural dissimilarity computation in the proposed model. Without any postprocessing, the proposed model based on the correlation of either of the two structural features outperforms 11 state-of-the-art saliency models on three saliency databases.

Tables

Table 1. Table 1: Performance comparison of the proposed SDS LC subscript SDS LC \text{SDS}_{\text{LC}} , SDS GM subscript SDS GM \text{SDS}_{\text{GM}} with other competing saliency models. The best and the second-best results are marked by red and blue colors, respectively.

	Toronto (120 images)				MIT (1003 images)				ImgSal (235 images)				Average
	EMD	AUC	CC	NSS	EMD	AUC	CC	NSS	EMD	AUC	CC	NSS	CC	NSS
IT [6]	2.837	0.799	0.503	1.297	5.146	0.767	0.334	1.093	2.043	0.799	0.577	1.325	0.391	1.151
GBVS [30]	2.237	0.830	0.600	1.519	4.310	0.821	0.421	1.366	1.777	0.833	0.674	1.564	0.480	1.414
JUDD [32]	3.037	0.839	0.563	1.389	5.329	0.835	0.419	1.329	2.706	0.837	0.628	1.429	0.468	1.352
LG [37]	3.443	0.753	0.386	1.072	5.764	0.746	0.292	0.987	3.213	0.696	0.373	0.883	0.314	0.977
CA [3]	3.041	0.781	0.465	1.272	5.304	0.755	0.313	1.052	2.374	0.791	0.577	1.384	0.353	1.110
SWD [9]	2.564	0.836	0.608	1.524	4.692	0.831	0.446	1.438	2.312	0.830	0.648	1.485	0.495	1.454
FES [13]	2.132	0.804	0.593	1.569	3.700	0.807	0.450	1.476	1.888	0.787	0.625	1.513	0.493	1.491
HFT [31]	2.283	0.825	0.606	1.631	4.304	0.809	0.422	1.395	1.709	0.821	0.695	1.670	0.486	1.463
SSD [21]	2.222	0.805	0.590	1.645	4.104	0.800	0.433	1.450	1.957	0.792	0.589	1.431	0.474	1.464
BMS [61]	3.115	0.786	0.518	1.560	5.348	0.785	0.374	1.314	2.625	0.788	0.620	1.599	0.429	1.385
LDS [7]	1.825	0.845	0.647	1.756	3.554	0.838	0.483	1.582	1.634	0.828	0.694	1.651	0.534	1.609
${\text{SDS}}_{\text{LC}}$	1.753	0.841	0.688	1.829	3.466	0.838	0.486	1.575	1.485	0.830	0.706	1.670	0.542	1.614
${\text{SDS}}_{\text{GM}}$	1.735	0.844	0.694	1.842	3.471	0.841	0.493	1.604	1.483	0.831	0.702	1.662	0.547	1.635

Table 2. Table 2: NSS Scores for each category of the Toronto database. The numbers of images in all categories are shown in brackets. The best and the second-best results are marked by red and blue colors, respectively.

	cat. 1	cat. 2	cat. 3	cat. 4	cat. 5	cat. 6	all
	(50)	(80)	(60)	(15)	(15)	(15)	(235)
IT	1.241	1.300	1.460	1.141	1.213	1.494	1.325
GBVS	1.406	1.553	1.648	1.538	0.790	1.618	1.564
JUDD	1.354	1.432	1.434	1.353	1.707	1.454	1.429
LG	0.793	0.961	0.897	0.507	0.915	1.051	0.883
CA	1.194	1.387	1.534	1.315	1.404	1.447	1.384
SWD	1.498	1.466	1.487	1.295	1.712	1.495	1.485
FES	1.541	1.590	1.249	1.486	1.975	1.629	1.513
HFT	1.512	1.607	1.727	1.764	2.114	1.761	1.670
SSD	1.329	1.466	1.510	1.070	1.612	1.454	1.431
BMS	1.597	1.542	1.680	1.436	1.732	1.620	1.599
LDS	1.621	1.617	1.634	1.509	2.078	1.695	1.651
${\text{SDS}}_{\text{LC}}$	1.582	1.702	1.659	1.554	1.994	1.634	1.670
${\text{SDS}}_{\text{GM}}$	1.605	1.687	1.649	1.499	1.988	1.611	1.662

Table 3. Table 3: Performance comparison of the model based on Eq.  9 with different normalization approaches and the proposed SDS LC subscript SDS LC \text{SDS}_{\text{LC}} .

$𝒩$	EMD	AUC	CC	NSS
0-1	3.275	0.780	0.439	1.100
MMLM [6]	2.274	0.781	0.442	1.109
graph [30]	2.794	0.792	0.519	1.345
${\text{SDS}}_{\text{LC}}$	1.753	0.841	0.688	1.829

Table 4. Table 4: Average Running Time over the Three Databases. All the competing models are implemented in Matlab, C, or Matlab + C.

Models	Implementation	Running time (s)
IT	M+C	0.23
GBVS	M+C	0.37
JUDD	M+C	12.07
LG	M+C	0.59
CA	M	32.70
SWD	M	1.00
FES	M	0.18
HFT	M	0.15
SSD	M	0.04
BMS	C	0.05
LDS	M	0.59
${\text{SDS}}_{\text{LC}}$	M	0.57
${\text{SDS}}_{\text{GM}}$	M	0.57

Table 5. Table 5: Performance comparison of the proposed SDS with features extracted in different color space and GM enabled / / disabled ( + ⁣ / ⁣ − +/- ). Specifically, SDS YIQ+GM subscript SDS YIQ+GM \text{SDS}_{\text{YIQ+GM}} is the original SDS GM subscript SDS GM \text{SDS}_{\text{GM}} .

	EMD	AUC	CC	NSS
${\text{SDS}}_{\text{YCbCr+GM}}$	1.758	0.843	0.687	1.826
${\text{SDS}}_{\text{YCbCr-GM}}$	2.136	0.820	0.579	1.479
${\text{SDS}}_{\text{RGB+GM}}$	1.834	0.837	0.641	1.623
${\text{SDS}}_{\text{RGB-GM}}$	3.267	0.750	0.403	0.971
${\text{SDS}}_{\text{YIQ-GM}}$	2.013	0.826	0.608	1.558
${\text{SDS}}_{\text{YIQ+GM}}$	1.735	0.844	0.694	1.842

Equations

$$\text{sim}=\frac{2\mathbf{f}(p)\cdot \mathbf{r}(p)+c}{\mathbf{f}(p{)}^2+\mathbf{r}(p{)}^2+c},$$

$$\mathbf{d}(p)=1-\text{sim}=\frac{(\mathbf{f}(p)-\mathbf{r}(p){)}^2}{\mathbf{f}(p{)}^2+\mathbf{r}(p{)}^2+c}.$$

$$\mathbf{d}(p)=|\mathbf{f}(p)-\mathbf{r}(p){|}^2.$$

$${\mathbf{d}}_i^x(p)=\frac{({\mathbf{f}}_i(p)-{\mathbf{r}}_i(p){)}^2}{{\mathbf{f}}_i(p{)}^2+{\mathbf{r}}_i(p{)}^2+c}.$$

$${\mathbf{D}}^x(p)={\left(\sum _{i=1}^K{\mathbf{d}}_i^x(p)\right)}^{\frac{1}{2}}.$$

$$\mathbf{S}(x)={\left(\sum _{p\in {\mathbf{D}}^x}\left({\mathbf{D}}^x(p)\cdot {\mathbf{W}}^x(p)\right)\right)}^{\theta }.$$

$$\begin{array}{rl}\mathrm{NSS}& =\frac{1}{{\sum }_{p}bool({\mathrm{FM}}_p)}\sum _p({\overline{\mathrm{SM}}}_p\times bool({\mathrm{FM}}_p)),\\ & \text{where }\overline{\mathrm{SM}}=\frac{\mathrm{SM}-\mu (\mathrm{SM})}{\sigma (\mathrm{SM})},\\ & bool({\mathrm{FM}}_p)=\{\begin{array}{ll}1,& \text{FMp>0}\\ 0,& \text{FMp=0}\end{array},\end{array}$$

$$\text{CC}=\frac{cov(\mathrm{SM},\mathrm{FM})}{\sigma (\mathrm{SM})\times \sigma (\mathrm{FM})},$$

$$S_{map}=\mathcal{G}\left(\sum _{i=1}^3\mathcal{N}({\mathbf{f}}_i)\right).$$

Full text

1 Introduction

The human visual system (HVS) can rapidly attend to salient regions from complex scenes. Saliency model, which aims to mimic this mechanism and predict where a human looks in an image, has attracted extensive research interest in computer vision. Regarding the human-fixation data under the free-viewing task when viewing images as the ground truth, the saliency models usually generate a saliency map with sparse blobs to predict the density of the fixation on the image. Such a saliency map can be used to benefit many vision-related applications, such as image quality assessment [1], video compression[2], and image retargeting[3].

A number of saliency models have been proposed in recent years. To detect the salient regions, inspired by the biological theories [4, 5], many saliency models highlight image regions that stand out from their surroundings. The distinctiveness-based models are the most representative models in this regard. The center-surround absolute difference on the biologically motivated features was first proposed by Itti et al.[6] and is widely used to detect the local distinctive regions [7, 8]. Later, by proposing a graph model to globally integrate the image regions, Harel *et al.*showed that the global distinctiveness could better predict saliency. Further, the spatially weighted global absolute difference of image patches in the principal components space [9] or CIELab color space [3, 10] also presents high accuracy in saliency detection. Besides, Rigas et al.[11] proposed to compute salience of the image patch by the distinctiveness based on the Hamming distance of sparse coding coefficients. By computing the probability of features, the local or global distinctive regions can also be effectively highlighted by the probability-based models [12, 13]. In addition, Hou *et al.*introduced the computation of saliency into the spectral domain. Although the spectral models usually have low complexity, they lack the biological plausibility [14]. The learning-based models have recently become popular owing to the powerful data-driven deep learning technology [15, 16, 17, 18, 19, 20]. However, their generalization ability needs further exploration [21]. Therefore, in this study, we focused on the distinctiveness-based models.

Moreover, another popular topic in computer vision, that is, perceptual image quality assessment (IQA) has also evoked great research interest in recent years. The goal of IQA is to predict the image quality consistent with the human subjective evaluation. When a pristine image is available, the IQA model type called the full reference IQA (FR-IQA) model computes the structural similarity between the given and pristine images, for example, the famous structural similarity metric (SSIM) [22]. The underlying assumption is that the HVS is sensitive to the structural change when evaluating the image quality. Specifically, the structural similarity measure is usually a point-by-point correlation of the common features, such as the local contrast (LC) [22] and gradient magnitude (GM) [23]. As both saliency detection and FR-IQA reflect how the HVS perceives the difference between image regions [24], the relationship between saliency and IQA has attracted much attention. In many IQA models, the saliency map is used to predict the importance of distortion to human perception. Specifically, the distortion in salient regions was proved to be perceived more annoying by Engelke et al.[25]. Then, some studies adopted the saliency map as weight in the computation of average similarity [26, 27, 28, 29, 1]. An interesting exception is the work proposed by Zhang et al.[24], who adopted the saliency maps as the feature maps for comparing the similarity between images. Overall, the migration from saliency detection to IQA widely exists in literature. However, to the best of our knowledge, no investigations have yet been conducted on the exploitation of IQA models for saliency detection.

This paper contributes toward saliency detection by exploiting IQA models. Many previous saliency models computed the absolute difference of features to measure the distinctiveness of image regions in the local [6, 8] or global extent [30, 9, 3]. However, according to the studies on IQA, the structural similarity is validated as a better similarity measure than the absolute difference in the context of HVS perception. Therefore, we propose using the structural dissimilarity as the distinctiveness measure between image patches for saliency detection. Specifically, the structural dissimilarity is computed as one minus the correlation of structural features, i.e., the relative difference of structural features. Then, the global structural dissimilarities of a patch to all the other patches are spatially weightedly summed to represent saliency of the patch. Besides, two widely used structural features, i.e, the LC and GM, extracted in the YIQ color space are adopted into the computation of the structural dissimilarity.

The proposed model has a similar framework (i.e., spatially weighted dissimilarity) to many other models [9, 3, 10]. In contrast to these models, the proposed model adopts a novel structural dissimilarity measure and intentionally employs a simple framework that excludes the post-processing or multi-scale operation. Based on the validation on three databases (i.e., Toronto [12], ImgSal [31], and MIT [32]), the proposed model outperforms 11 state-of-the-art saliency models. The effectiveness of the proposed model and the adopted similar but simpler framework indicate the high relevance of the proposed structural dissimilarity and the allocation of fixation under the free-viewing task when viewing images, thereby highlighting the novelty of the proposed model.

Furthermore, the adopted LC in the YIQ color space is similar to the center-surround contrast of the colors and intensity in Itti’s model. However, Itti *et al.*summed the maximum-mean-local-maxima (MMLM) normalized feature maps, whereas we summed the patch-based global relative differences of features as the salience. The better performance of the proposed model implies that the proposed method would be a better normalization approach for highlighting the salient regions. A further experimental comparison verifies this claim.

Moreover, a comprehensive analysis was conducted to study the effectiveness of the proposed model in terms of three aspects: the spatially weighted dissimilarity-based framework, the structural features, and the correlation-deduced relative difference measure.

The contributions of this paper are summarized as follows:

1. Structural dissimilarity is introduced into saliency detection for the first time.
2. The proposed model is highly competitive with the state-of-the-art models.
3. The global structural dissimilarity is more effective in highlighting salient regions than the widely used normalization approaches.
4. A comprehensive analysis was conducted to better understand the reasons for the good performance of the proposed model.

The remainder of the paper is organized as follows. Section 2 introduces the related work and background. Particularly, a brief introduction of IQA is presented to bridge the knowledge gap between saliency detection and IQA. Section 3 presents the proposed model in detail. Section 4 presents the experimental validation. Section 5 presents the comprehensive analysis on the effectiveness of the proposed model. Finally, Section 6 concludes the paper.

2 Related Work and Background

2.1 Related Saliency Models

Distinctiveness-Based Models. In distinctiveness-based models, the image regions with distinctive features in the local surroundings or global extent are regarded as salient regions. Itti et al.[6] proposed a representative work of distinctiveness-based models. Specifically, they decomposed an image into a set of feature channels, i.e., the colors, intensity, and orientations of the luminance. Then, the center-surround local absolute difference was used to extract the local distinctiveness map in each feature channel. Finally, the distinctiveness maps of all the channels were integrated as the saliency map. This framework inspired many saliency models. For example, Fang et al.[7] computed the flexible center-surround difference in the principal-component-based subspaces of the image. Ishikura et al.[8] computed the local extreme values in the CIELab color space. Instead of the local distinctiveness, Harel et al.[30] proposed the use of a graph to globally interrelate the image regions. The spatially weighted absolute difference of features between image regions was set as the edge between nodes in the graph. Then, the global distinctive features were highlighted through a random walker process. Goferman et al.[3], and Duan et al.[9], computed a similar spatially weighted difference between image patches. However, unlike that the work of Harel et al.[30], the patch’s differences to all the other patches are simply summed to represent the global distinctiveness. Further, Goferman et al.[3] computed the difference in the CIElab color space, while Duan et al.[9] computed the difference in the dimension-reduced principal component space.

Based on similar scheme, Zhou and Jin [10] proposed a multi-scale operation to obtain the final saliency map.

In addition to computing the difference of features, the probability-based approach is a popular method to measure the distinctiveness of features. Principal component analysis (PCA) and independent component analysis (ICA) are widely used to extract structure-related components according to image patches. Based on the statistics of the PCA or ICA coefficients on the objective image (e.g., Refs. 33, 34, 35), or on a set of natural images (e.g., Refs. 36, 12, 37), the detection of the distinctive features becomes a concept of rarity. Specifically, a patch is more likely to be salient if it has rare PCA- or ICA-based features.

Sparsity-Based Models By assuming that an image can be represented in a redundant part and a sparse salient part [38, 39, 40], many sparsity-based models are based on the low-rank matrix recovery theory [41]. Particularly, Wang et al.[40] made an in-depth analysis of the sparse codes and the sparse reconstruction error that yields an effective saliency model. Besides, similar to the distinctiveness-based model, Rigas et al.[11] computed the saliency map via measuring the global distinctiveness of patches based on Hamming distance of sparse coding coefficients.

Spectral Models. The first spectral saliency model was proposed by Hou et al.[42], who suggested using the spectral residual of the amplitude spectrum to obtain the saliency map. Later, Guo et al.[43] suggested that the phase spectrum is the key for saliency detection. Li et al.[31] proposed a hypercomplex Fourier transform with a multiscale strategy for saliency detection. Li et al.[21] developed a learning-based filter with respect to phase spectra to minimize the difference between the phase spectrum of the input image and fixation density map.

Learning-Based Models. The learning-based detectors of the semantic cues, e.g., face, human, and car, are proved to be beneficial for saliency detection [32, 44]. Moreover, benefiting from the powerful deep neural network, many recent saliency models achieved excellent performance in fixation prediction, e.g., Refs. 15, 16, 17, 18, 19, 20.

The Post-Processing. Beyond the saliency-detection algorithms, post-processing could further enhance the performance of saliency detection. For example, the fixation is more likely to be located at the image center [45]. Thus, the border cut and center post-weighting that can highlight the center of the image will benefit saliency detection [36, 46, 9, 3]. However, the center regions will not always be salient. Moreover, many saliency models would over-highlight the boundaries of the salient regions. The Gaussian post-blurring, as described in Refs. 6, 30, 7, can then extend the large values from the boundaries to the inner regions. However, it will simultaneously highlight the unsalient exteriors.

Beyond Static Images. Saliency detection in the static images is the focus of this study. In addition, saliency detection in videos and 360° (omnidirectional) images/videos has attracted significant research interests. In particular, viewers usually do not have a complete view of the 360° content. Thus, the corresponding head rotation poses a problem for saliency detection [47]. Moreover, when shown in equirectangular projection, the 360° contents are non-Euclidean [48]. To address these problems, many studies on 360° images/videos in viewing databases [49, 50], image quality assessment [51], and saliency detection [52], have been conducted in recent years.

2.2 Image Quality Assessment

To bridge the knowledge gap, a brief introduction of IQA is presented. Based on the availability of the pristine image, the IQA models can be classified into FR models, reduced reference models, and no reference models. Our study focuses on the FR models because we use this type of model to measure the structural similarity between two patches. Fig. 1 shows a typical framework of the structural-similarity-based FR-IQA models. Generally, the IQA models decompose the images into various local-feature maps. As in many saliency models, the features in the IQA models are typically common in computer vision, for example, the features that could capture the local structures (i.e., the structural features) and the chrominance features. The point-by-point similarities in different maps can be computed and combined to obtain a local quality map. Then, a pooling step, such as the mean average or standard deviation, is executed on the local quality map to obtain the overall quality score of the degraded image.

Many IQA models follow this framework. For example, the well-known SSIM [22] extracts three features of images: local mean, LC, and local structure. Then, the point-wise similarity comparisons of the luminance, contrast, and structures are computed, combined, and pooled to yield the IQA model. The main motivation to use the structural similarity is that the HVS is highly sensitive to the local structure. This idea inspired many IQA models. Xue et al.[23] proposed the use of the GM as the only structural feature for measuring structural similarity. Specifically, the GM is the root mean square of image directional gradient along the horizontal and vertical directions. Some works (see 53, 54) validated that the Laplacian-of-Gaussian (LOG) signal acts as an effective and efficient structural feature for IQA. Moreover, the feature similarity metric [55] used the phase congruency and GM of the Y component and the pixel value of IQ chrominance components in the YIQ color space. The saliency map is used in the visual saliency-induced index [24] as a feature map, together with the GM of L component, and the pixel value of the MN chrominance components in the LMN color space.

To measure the similarity between the feature maps, the introduced IQA models mainly used a correlation-based measure as follows:

[TABLE]

where $\bm{f}$ and $\bm{r}$ are the feature maps of the original and degraded image, respectively, $p$ denotes the point-by-point computation for pixel $p$, and $c$ is a small positive constant.

The above similarity measure can simply be deduced as a dissimilarity measure, i.e.,

[TABLE]

This dissimilarity measure can be regarded as the normalized version of the squared absolute difference, i.e.,

[TABLE]

The denominator $\bm{f}(p)^{2}+\bm{r}(p)^{2}+c$ makes Eq. 2 a relative dissimilarity measure, which is claimed to be consistent with Weber’s law in Ref. 22. In IQA models, parameter $c$ is used to avoid division by zeros [22]. In this study, we show that parameter $c$ serves as a normalization parameter to control the effects of the insignificant features in the proposed model. Only when the energy of the comparing features (i.e., $\bm{f}(p)^{2}+\bm{r}(p)^{2}$) approaches or exceeds the level of $c$, can their absolute difference affect the result computed by Eq. 2 (for details, refer to Sec. 5). The comprehensive analysis of the advantage of the correlation-based dissimilarity measure can be found in Refs. 56, 57.

3 Method

In this paper, we propose the use of structural dissimilarity induced by IQA models as the distinctiveness measure for saliency detection. To consider computation complexity, the comparison between patches was conducted in a nonoverlapped manner. To facilitate nonoverlapping of the split, we resized the image to multiples of patches $[Mw\times Nw]$, where $w$ is the width of the patch in pixels, $M$ and $N$ are the numbers of patches in height and width of the image. We experimentally set $w=24$, and $\mathrm{min}(M,N)=11$. Fig. 2 illustrates the process to compute the saliency of patch $x$, which is marked with a red box on the resized image $\bm{I}$. The process involves four main stages.

1. Feature Extraction. To compute the structural dissimilarity between image patches, $\bm{I}$ is first decomposed into a set of structural features. According to studies, such as Refs 8, 58, 6, both the luminance and chrominance information are crucial for saliency detection. Therefore, structural features are proposed to be extracted in each color channel of the YIQ color space. Suppose there are $K$ channels, then the feature maps are indicated by $\bm{f}_{i}$ ($i=1,2,...,K$).

2. Dissimilarity Measure. The correlation deduced relative difference measure (i.e., Eq. 2) is proposed to compute the point-wise dissimilarity between feature patches. In each feature channel, we duplicated patch $x$ in a nonoverlapped manner to obtain duplication feature map $\bm{r}_{i}^{x}$, as shown in Fig. 2. Then, the global comparison of $x$ with all other patches is equivalent to the comparison between $\bm{r}_{i}^{x}$ and $\bm{f}_{i}$. We rewrite the computation of dissimilarity map $\bm{d}_{i}^{x}$ as

[TABLE]

3. Combination. The dissimilarity maps in different feature channels are combined to an overall dissimilarity map $\bm{D}^{x}$. Specifically,

[TABLE]

4. Spatial Weighting and Sum. Spatially weighted $\bm{D}^{x}$ are summed to predict the saliency of $x$. This spatial weighting has been adopted in many works (e.g., Refs 30, 3, 9) and controls the extent of the global comparison. Specifically, spatial weighting map $\bm{W}^{x}$ is the distance of patch $x$ to other patches. Let $\mathbb{x}$ and $\mathbb{y}$ indicate the location of patch $x$ and $y$, respectively. The spatial distance between $x$ and $y$ is computed by $\textrm{exp}(-\frac{\|\mathbb{x}-\mathbb{y}\|^{2}}{2\sigma^{2}})$, where $\sigma^{2}$ is a scale parameter. To ensure the same size for $\bm{W}^{x}$ and $\bm{D}^{x}$, the points in $y$ share the same distance value. Finally, the spatially weighted global dissimilarity of $x$ (i.e., the saliency of $x$) is summed as

[TABLE]

The global sum of the relative differences would result in a slight difference between the saliency of patches. Therefore, this paper proposes the exponential operation by using exponent $\theta$ to highlight the salient patches. We empirically set $\theta=8$. It is worth noting that the relative values in the saliency map will not be changed.

The abovementioned process was repeated until the saliency of all the patches is obtained. Then, the saliency map was normalized to the dynamic range of $[0,1]$. No other post-processing operation, e.g., Gaussian post-blurring or center post-weighting, was conducted.

4 Experiments

4.1 Experimental Protocols

4.1.1 Features

The structural dissimilarity based on the correlation of the structural features is crucial in the proposed framework. To validate the effectiveness of the proposed framework, we adopted two widely used structural features: LC and GM. Let the structural-dissimilarity-based saliency (SDS) denote the proposed framework. The two corresponding models are then denoted as $\text{SDS}_{\text{GM}}$ and $\text{SDS}_{\text{GM}}$, respectively.

4.1.2 Databases

The performance of the saliency models depends on the consistency between the saliency map and the corresponding eye fixations. Three popular saliency databases were used in this study: Toronto [12], MIT [32] and ImgSal [31]. The Toronto database consists of 120 images with human fixation data recorded on 20 viewers, while the MIT database consists of 1003 images and the corresponding eye fixation data was recorded on 15 viewers. Both Toronto and MIT databases have natural indoor and outdoor scenes. However, MIT contains many daily-life and portrait pictures, making it a more challenging database. The ImgSal database consists of 235 images and the corresponding eye fixation data was recorded on 21 viewers. The images in ImgSal were classified into six categories: 50/80/60 images with large/intermediate/small salient regions, respectively, 15 images with a cluttered background, 15 images with repeating distractors, and 15 images with both large and small salient regions.

4.1.3 Evaluation metrics

Many evaluation metrics exist for the evaluation of saliency prediction performance. According to the comprehensive study on the evaluation metrics by Bylinskii et al.[59], the normalized scanpath saliency (NSS) and the linear correlation coefficient (CC) are the most reasonable metrics, and are recommended to be selected.

The NSS computes the mean value at the fixation locations on a z-score-normalized saliency map. Therefore, the larger NSS scores are desirable. Let $\mathrm{SM}$ denotes the saliency map of a given image, and $\mathrm{FM}$ denotes the corresponding fixation map (i.e., the eye fixation points of the image). The NSS score can be computed as

[TABLE]

where $\mu$ and $\sigma$ compute the mean and standard deviation of the map, respectively, and $p$ denotes the point-wise operation.

The CC computes the Pearson’s linear correlation coefficient between the saliency map and Gaussian-blurred fixation map (i.e., the fixation density map). We still use $\mathrm{FM}$ to denote the Gaussian-blurred fixation map. The CC score can be computed as

[TABLE]

where $cov$ indicates the covariance computation. The CC has a value in the range of $[-1,+1]$, where $\text{CC}=1$ is the total positive linear correlation.

In addition to the NSS and CC scores, in the following experiments, we report on the scores of two historically widely used metrics: the earth mover’s distance (EMD) and area under the ROC curve (AUC) [32]. A lower EMD and higher AUC indicate a better consistency between the saliency and fixation maps. In the four evaluation metrics, the recommended NSS has the lowest computation complexity. Therefore, in the following analysis of the effects of the parameters and dissimilarity measure, we only report on the NSS scores. Then, in the comprehensive comparison with other competing saliency models, we report the scores evaluated by all the four metrics. Specifically, we adopted the codes of the four evaluation metrics provided by Ref. 60.

4.1.4 Competing saliency models

To demonstrate the performance of the proposed SDS, we compared it with 11 state-of-the-art and representative saliency models, including IT [6], GBVS [30], JUDD [32], SWD [9], CA [3], FES [13], HFT [31], SSD [21], BMS [61], and LDS [7]. All the source codes are publicly available on the Internet. We used the default parameters provided in the source codes for all models. In these models, HFT and SSD are the spectral saliency models, JUDD and LDS are the learning-based models, and the other models are distinctiveness-based models. Moreover, IT, GBVS, FES, HFT, BMS, and LDS adopt the Gaussian post-blurring, whereas JUDD, SWD, and CA adopt the center post-weighting.

The saliency maps generated by the various saliency models differed in size. Thus, a resizing process was embodied in the codes of the evaluation metrics provided by Ref. 60. All the saliency maps were cubic-interpolated to the same size as the fixation map. In addition, all the saliency maps were normalized to the dynamic range of $[0,1]$ before evaluation.

4.2 Effects of the Parameters

There are two important parameters in the proposed model: normalization parameter $c$ in the structural dissimilarity computation (i.e., Eq. 5) and scale parameter $\sigma^{2}$ in the spatial weighting computation (i.e., Eq. 6). Normalization parameter $c$ controls the effects of the insignificant features. For comprehensive analysis, we report the saliency detection performance with respect to parameter $c$ over the three adopted databases of $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ in terms of the NSS score. The results are shown as scatter plots in Fig. 3. As shown, considering the wide range of $c$, both $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ are not very sensitive to the changing of $c$. However, $\text{SDS}_{\text{LC}}$ is more insensitive because feature LC usually possess a larger value than GM. In addition, the best NSS scores of $\text{SDS}_{\text{GM}}$ on the three databases are achieved when $c=20$. Similar results can be observed for $\text{SDS}_{\text{LC}}$, except on the Toronto database, for which the best NSS score is achieved when $c=2000$. Considering the overall performance on the three databases, we set $c=20$ for both $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ in the following experiments.

Scale parameter $\sigma^{2}$ in the spatial weighting computation controls the extent of global comparison, which in turn affects the ability to detect various sized salient regions. Taking advantage of the ImgSal database containing categories with various sized salient regions, we analyzed the detection performance of $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ under different $\sigma^{2}$. The specific NSS scores against $\sigma^{2}$ are shown in Fig. 4. In particular, we also report the detection performances on categories 1, 2, and 3, in which the sizes of salient regions showed a decreasing trend.

Fig. 4 shows that the NSS scores are almost the same when $\sigma^{2}=3\text{ to }6$ on the whole database. By contrast, apparent changes can be observed at both small and large $\sigma^{2}$. In addition, the NSS scores for categories 1 and 2 (i.e., large and intermediate salient regions, respectively) are greatly increased at the beginning of the increase of $\sigma^{2}$. However, the NSS scores for category 3 (i.e., small salient regions) are almost the same at various $\sigma^{2}$. This is because the small salient region will be dissimilar to both the local and the global surroundings, as shown in the first row of Fig. 5. For a large salient region, the dissimilarity measure in the local extent will tend to highlight the boundaries of the region. By contrast, its inner region, especially the inner homogeneous region, will be highlighted by a global dissimilarity measure, as shown in the second row of Fig. 5. However, a slight performance drop can also be observed at the large $\sigma^{2}$. This is because the unsalient regions might be incorrectly highlighted by the global dissimilarity measure, as illustrated in the last row of Fig. 5. When measured in the global extent, the patches at the sky, which are distinct to the vast grass and trees, are highlighted on a large $\sigma^{2}$.

Considering the scatter plots in Fig. 4, the best NSS scores are reached at $\sigma^{2}=4,3,2$ for categories 1,2, and 3, respectively, both for $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$. Therefore, considering the overall performance on different sized salient regions, we fixed $\sigma^{2}=3$ for both $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ in the following experiments.

4.3 Comparison with the State-of-the-art models

The mean scores evaluated by all four metrics on the three databases were calculated to compare the proposed $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ with other competing saliency models. Table 1 summarizes the results.

The scores of the proposed $\text{SDS}_{\text{GM}}$ and $\text{SDS}_{\text{LC}}$ can be observed to be the same on the three databases. $\text{SDS}_{\text{GM}}$ outperforms all the other competing models with respect to the EMD, CC, and NSS scores on the Toronto and MIT databases. $\text{SDS}_{\text{GM}}$ achieves the best CC and NSS scores on ImgSal. Furthermore, LDS achieves the best AUC score on Toronto and the second-best AUC and NSS scores on MIT. JUDD and GBVS achieve the best and second-best AUC scores on ImgSal, respectively. For better comparison of the performance, we calculated the weighted average of CC and NSS scores over the three databases. The average weights are determined according to the number of images in the three databases. Table 1 summarizes the results. The table shows that $\text{SDS}_{\text{GM}}$, $\text{SDS}_{\text{LC}}$, and LDS achieve the top 3 positions, among which the $\text{SDS}_{\text{GM}}$ is the best.

For statistical significance test of mean scores in terms of NSS and CC, We further conducted the t-test at $p\leq 0.05$ level of significance [62]. The results are summarized in Fig. 6, where a value of ‘1’ indicates the model in row has a larger mean score than the model in the column and the two mean scores are statistically significantly different. Moreover, we have checked the normality distribution assumption of the scores by the Jarque-Bera test [63]. If a model violates this assumption, the corresponding results in Fig. 6 are colored in gray. Here, $\text{SDS}_{\text{GM}}$ is tested as the representative, because the proposed $\text{SDS}_{\text{GM}}$ and $\text{SDS}_{\text{LC}}$ have similar performance.

It can be seen that on the Toronto database, the proposed model is significantly better than the other models in terms of both NSS and CC. Similar results can be observed on the MIT database in terms of CC. And the proposed model and LDS perform well with no significant difference in terms of NSS. On the ImgSal database, the proposed model and HFT perform well with no significant difference in terms of both NSS and CC. Therefore, we can conclude that the proposed model is highly competitive to these advance models.

To further validate the ability of the proposed models in handling the different-sized salient regions, we compared all the models over each category of ImgSal. Table LABEL:tab:imgsalcate summarizes the corresponding NSS scores. As shown, the HFT model achieves the best scores for categories 3–6. Furthermore, LDS presents a promising performance in each category. Note that the HFT model conducted a multiscale analysis in the spectral domain and designed a criterion to select the optimal scale that can best highlight the salient regions. LDS conducted a flexible-scaled random contrast to highlight the distinctive features. Therefore, the multiscale operation is a positive solution to handle different sized salient regions. In contrast, the proposed $\text{SDS}_{\text{GM}}$ and $\text{SDS}_{\text{LC}}$, without the multiscale operations, also present competitive results in all categories. This effectiveness is due to the following two aspects. First, the relative difference with normalization parameter $c$ can highlight the inner regions of the large salient regions. Second, proper scale parameter $\sigma^{2}$ facilitates the detection of various sized salient regions.

The qualitative comparison is shown in Fig. 7. The saliency maps of $\text{SDS}_{\text{GM}}$ are used to represent the proposed model. As shown, the saliency maps of the recent models are more visually consistent with the ground-truth fixation density maps. By contrast, the early works tend to highlight the local distinctive regions. Moreover, among all the saliency models, although $\text{SDS}_{\text{GM}}$ is the most stable and accurate to predict the fixations in the examples, it only computes the global distinctiveness of the simple feature: GM. The portrait images in the last two columns of Fig. 7 are still complicated cases for $\text{SDS}_{\text{GM}}$.

4.4 Comparison with Other Normalization Approaches

From the perspective of normalization, the features in the proposed model is normalized by the patch-based global comparison of relative difference (i.e., the global structural dissimilarity), whereas the features in Itti’s model is normalized by the MMLM. By computing the difference between the maximum and the mean of local maxima, MMLM can promote the feature maps with a small number of strong peaks. However, in contrast to the proposed measure, MMLM cannot capture the difference of local structures to which the HVS is highly sensitive.

For comparison, according to Itti’s model, the proposed model based on Eq. 6 can be modified as

[TABLE]

In Eq. 9, compared with Eq. 6, the patch-based global comparison of relative difference is replaced by the MMLM-based normalization (denoted by $\mathcal{N}$); the exponential operation is removed; the Gaussian blurring $\mathcal{G}$ with the standard deviation set as $5\%$ of the width of $S_{map}$ is conducted; and $\bm{f}_{i}$ is set as the LC feature.

In addition to MMLM, another two widely used normalization approaches, i.e., the 0-1 normalization and the graph-based approach [30], are adopted as alternatives of $\mathcal{N}$. The 0-1 normalization scales the feature maps to a fixed dynamic range $[0,1]$. The graph-based normalization also computes the spatially weighted global difference, and is conducted on the value of features.

The models based on the three different normalization approaches are tested on the Toronto database. The results shown in Table 3 support the claim that the proposed approach is better than other normalization approaches in the highlighting of the salient regions.

4.5 Complexity Analysis

The proposed model involves two key stages, i.e., the structural feature extraction and the patch based global relative difference computation. In the first stage, the bicubic interpolation of pixels based image resizing and the pixel-based RGB$/$YIQ should be conducted on an image first. Then, the convolution-based gradient computation or Gaussian filtering is needed to obtain the GM or LC features. Note that the number of pixels in the resized image is $M\!w\!\times\!N\!w$ (denoted by $\!\Omega$). Thus, the time complexities of image resizing, color space conversion, and convolution are all $O(\!\Omega\!)$. In the second stage, the salience of each image patch is computed using Eq. 6. Specifically, each patch is compared with other patches point-by-point using Eq. 4 in three feature channels. Thus, it requires $3\!\times\!\Omega$ times calculation of Eq. 4, $3\!\times\!\Omega\!-\!1$ times of additions, $\Omega$ times of multiplications, and one exponential operation to compute the salience of a patch. Thus, when computing the salience of all the patches, the total time complexity of the second stage is $O(\Omega\times({M\!\times\!N}))$.

Table 4 shows the running time comparison between the proposed models and the other 11 saliency models. All the models were run on a computer with 4.00 GHz CPU and 16G RAM. With Matlab implementation, the proposed $\text{SDS}_{\text{LC}}$ and $\text{SDS}_{\text{GM}}$ take approximately 0.5 s to generate a saliency map based on the average time cost on the three databases. The running time of the proposed models was moderate in all the competing models. The low running time is because all the operations except the feature extraction in the proposed framework are conducted in a point-wise manner.

5 Analysis of the Proposed SDS Model

In this section, we seek to identify the reasons for the good performance of the proposed model in terms of three aspects: 1) the framework (i.e., spatially weighted dissimilarity); 2) the structural features; 3) the correlation-deduced relative difference. Specifically, aspects 2) and 3) constitute the proposed structural dissimilarity measure.

5.1 Framework

The proposed model is based on a standard spatially weighted dissimilarity-based framework, and thus, it has the same advantage as other models based on this framework. Specifically, the spatial weighting implicitly assigns larger weights to the center patches [30]. The implicit center bias is significant in addressing the photographer bias and the central fixation tendency of the observers [45]. For the proposed model, if the structural dissimilarity is pruned (i.e., the dissimilarity $\bm{D}^{x}(p)$ is directly set as one in Eq. 6), the remainder is the implicit center bias. Based on the experiments, the corresponding NSS scores of the remainder model (1.274, 1.279, and 1.225 on the Toronto, MIT, and ImgSal databases, respectively) are good. Therefore, the first reason for the good performance of the proposed model is that the adopted framework involves an implicit center bias.

5.2 Structural Features

The proposed dissimilarity is measured based on the structural features, which are extracted by the GM/LC in the YIQ color space. The structural features are crucial for the proposed model, because an intensity-color-opponent-based (ICO) color space can highlight the salient regions with distinct colors. Then, the structure extractors GM or LC can make the salient regions stand out further.

To verify the effects of the structural features, the YIQ color space was replaced by each of YCbCr and RGB, of which the YCbCr is also an ICO color space; the structure extractor was disabled so that the dissimilarity could be directly measured based on the colors. In the experiments, GM was used as the representative structure extractor. The results obtained on the Toronto database are summarized in Table 5. When the GM is enabled, the proposed model shows similar performance on the YIQ and YCbCr color space. However, the performance on the RGB color space is obviously inferior. When the GM is disabled, the performance declines significantly for all the color spaces. Therefore, we can conclude that both the structure extractors and color space are important for the proposed model.

5.3 Correlation-deduced relative difference

In contrast to the absolute difference, the relative difference can limit the value of the difference. Therefore, the global difference (i.e., saliency as in Eq. 6) will not be greatly affected by individual excessive differences. The dissimilarity in the organization of the structural features is then highlighted. Fig. 8 illustrates the residue maps computed based on the absolute difference and relative difference for three example patches. The mean difference is shown under the corresponding residue map. The red-marked patch only possesses small area of boundaries. However, the sharp boundaries still lead to a large average absolute difference. By contrast, the difference is limited by the relative difference measure. Fig. 8 also shows the importance of the normalization parameter $c$. Although the relative difference can limit the large absolute difference, the difference between insignificant features would simultaneously be enhanced. For example, when $c=0.01$, the background patch (marked by purple box) shows the largest mean relative difference. However, when $c=30$, the differences between the insignificant features are obviously suppressed. Specifically, an obvious change occurs in the background regions.

To further validate the advantage of the correlation deduced relative difference, we replaced the relative dissimilarity measure (Eq. 4) by the absolute difference (Eq. 3) in the proposed model.

The corresponding model with tuned parameters is tested on the Toronto database. The results evaluated on the Toronto database are 2.350, 0.818, 0.561, and 1.453 in terms of EMD, AUC, CC, and NSS, respectively. It can be seen that the scores are significantly reduced compared with the results of the original model as in Table 1. Therefore, we can conclude that the relative difference significantly improves the performance of the proposed model in saliency detection compared with the normally used absolute difference.

Based on the above analysis, all the three aspects attribute to the good performance of the proposed model.

6 Discussion and Conclusion

In this study, the image quality assessment (IQA) was introduced into saliency detection. We proposed the use of the structural dissimilarity induced by the IQA models as the dissimilarity measure between image patches. To highlight the effectiveness of the proposed structural dissimilarity, we adopted a simple spatially weighted dissimilarity framework that excludes the multi-scale operation or post-processing. Compared with the other 11 state-of-the-art saliency models, the proposed model presented highly competitive saliency detection performance tested on three saliency databases. Based on the results of the comprehensive experiments in Section 5, the structural features in the color space and the relative difference based measure (i.e., the proposed structural dissimilarity) attribute to the high performance of the proposed model. Based on the computation of the spatially weighted structural dissimilarity, the regions with the distinctive structures are highlighted. According to the biological theories [4, 5], image regions that stand out from their background are prioritized at almost all levels of the visual system and will attract human attention. Therefore, the highlighting of the structurally distinctive regions based on the proposed model is consistent to the allocation of human fixation.

Besides, the fixation map is generated by summing up the fixation points of a number of viewers during eye scanning on an image. In most works of saliency prediction including ours, the benchmark is the fixation map and not the eye scanpaths. In the works such as Ref. 6, the inhibition-of-return scheme is adopted as the post-processing on saliency map to predict the eye scanpaths on an image. Then, the eye scanpaths are predicted according to the descending salient values. However, besides the salient values, the eye scanpaths also depend on the context of the image content and the prior information [64]. Therefore, an updating saliency map between saccades considering such a dependence of context and prior is expected to benefit the prediction of eye scanpaths. In future works, we will continue to investigate the prediction of eye scanpaths.