A Perceptually Weighted Rank Correlation Indicator for Objective Image Quality Assessment

TL;DR

This paper introduces a perceptually weighted rank correlation indicator for objective image quality assessment that emphasizes high-quality images and perceptually similar pairs, providing a more accurate evaluation of IQA algorithms.

Contribution

It proposes a novel weighted rank correlation measure that accounts for perceptual importance and subjective uncertainty, improving the assessment of IQA metrics.

Findings

Better correlation with human perception in IQA tasks

Enhanced ability to rank high-quality images accurately

Robustness across degraded and enhanced images

Abstract

In the field of objective image quality assessment (IQA), the Spearman's $\rho$ and Kendall's $\tau$ are two most popular rank correlation indicators, which straightforwardly assign uniform weight to all quality levels and assume each pair of images are sortable. They are successful for measuring the average accuracy of an IQA metric in ranking multiple processed images. However, two important perceptual properties are ignored by them as well. Firstly, the sorting accuracy (SA) of high quality images are usually more important than the poor quality ones in many real world applications, where only the top-ranked images would be pushed to the users. Secondly, due to the subjective uncertainty in making judgement, two perceptually similar images are usually hardly sortable, whose ranks do not contribute to the evaluation of an IQA metric. To more accurately compare different IQA algorithms, we explore a perceptually weighted rank correlation indicator in this paper, which rewards the capability of correctly ranking high quality images, and suppresses the attention towards insensitive rank mistakes. More specifically, we focus on activating `valid' pairwise comparison towards image quality, whose difference exceeds a given sensory threshold (ST). Meanwhile, each image pair is assigned an unique weight, which is determined by both the quality level and rank deviation. By modifying the perception threshold, we can illustrate the sorting accuracy with a more sophisticated SA-ST curve, rather than a single rank correlation coefficient. The proposed indicator offers a new insight for interpreting visual perception behaviors. Furthermore, the applicability of our indicator is validated in recommending robust IQA metrics for both the degraded and enhanced image data.

Figures

Equations

$$\Gamma =\frac{{\Sigma }_{i,j=1}^n{a}_{ij}{b}_{ij}}{\sqrt{{\Sigma }_{i,j=1}^n{a}_{ij}^2{\Sigma }_{i,j=1}^n{b}_{ij}^2}}$$

$$a_{ij}$$

$$b_{ij}$$

$$\rho =\frac{{\Sigma }_{i,j=1}^n(p_i-p_j)(q_i-q_j)}{\sqrt{{\Sigma }_{i,j=1}^n(p_i-p_j{)}^2}\sqrt{{\Sigma }_{i,j=1}^n(q_i-q_j{)}^2}}$$

$$\rho =1-w{\Sigma }_{i=1}^n{d}_i^2$$

$$a_{ij}$$

$$b_{ij}$$

$$\tau =w\sum _{i\ne j=1}^n\text{sgn}(p_i-p_j)\text{sgn}(q_i-q_j)$$

$$\text{sgn}(p_i-p_j)=\{\begin{array}{ll}1,& p_i-p_j>0\\ -1,& p_i-p_j<0\end{array}$$

$$S(\text{x},\text{y},T)=f[A(\text{x},T),D(\text{p},\text{q}),M(\text{p},\text{q})]$$

$$A(\text{x},T)=\frac{1}{1+e^{-C_1(\Vert x_i-x_j\Vert -T)}}$$

$$\frac{1}{1+e^{-C_1\cdot 2{\sigma }_i}}=0.95$$

$$\hat{\text{x}}=\{\begin{array}{ll}(\omega \text{x}+\epsilon )\times 100,& \text{x is MOS}\\ [1-(\omega \text{x}+\epsilon )]\times 100,& \text{x is DMOS}\end{array}$$

$$\hat{\mathbf{\sigma }}=\omega \times \mathbf{\sigma }\times 100$$

$$D(\text{p},\text{q})=\text{sgn}(p_i-p_j)\text{sgn}(q_i-q_j)$$

$$M(\text{p},\text{q})=\frac{e^{d_{ij}}+e^{l_{ij}}-2}{{\sum }_{i\ne j=1}^n(e^{d_{ij}}+e^{l_{ij}}-2)}$$

$$S(\text{x},\text{y},T)=\sum _{i\ne j=1}^{n}A(\text{x},T)\cdot D(\text{p},\text{q})\cdot M(\text{p},\text{q})$$

$$w[D(\text{p},\text{q})]=\{\begin{array}{ll}1,& D(\text{p},\text{q})<0\\ 0,& \text{otherwise}\end{array}$$

$$△d_N=\frac{1}{N}\sum _{i=n+1-N}^n{x}_{q_i}-\frac{1}{n-N}\sum _{j=1}^{n-N}{x}_{q_j}$$

$$△\text{MOS}=\frac{1}{n-1}\sum _{N=1}^{n-1}△d_N$$

$${\text{AUC}}_{ca}={\int }_{T_{min}}^{T_{max}}S(\text{x},\text{y},T)dT$$

$$\Delta {\overline{\text{MOS}}}_k=\frac{1}{L}\sum _{l=1}^L\Delta {\text{MOS}}_{k,l}$$

$${\overline{\text{PWRC}}}_k=\frac{1}{L}\sum _{l=1}^L{\text{PWRC}}_{k,l}$$

Full text

I Introduction

In most image processing applications, human eyes are the ultimate receiver of the visual signal, where the subjective test could offer ideal evaluations for various process results. However, confronting the massive image data generated in our daily life, it has become impossible to manually annotate the image quality via substantial human subjects, which is usually time consuming, troublesome and expensive [1]. As an alternative, it is more economical and practical to count on well-designed objective image quality assessment (IQA) metrics.

Given multiple processed or contaminated images, a main target of objective IQA is to correctly sort their quality levels via a computational model, which could approach the rank-order produced by the subjective judgement. A reliable objective IQA model not only serves as an image quality monitor, but also plays important roles in optimizing various quality driven applications, such as, image/video coding [2], image fusion [3], contrast enhancement [4], and so on. With the boom of perceptually friendly image/video processing systems [5, 6, 7], recent decades have witnessed the growing interests in the development of IQA algorithms. A large number of subject rated databases (such as, LIVE II [8], TID2013 [9], CSIQ [10], ChallengeDB [11]) and objective metrics (including the full-reference [12], reduced-reference [13], and no-reference models [14, 15]) are successively proposed to interpret the human perception of image quality from different perspectives. Thanks to the efforts of these researches, many exciting findings and technologies are verified efficient in modeling the perceptual image quality, such as, the perceptual visibility threshold [16, 17], the visual attention psychology [18, 19], the structural similarity measurement [20, 21, 22], the natural scene statistics [23, 24], the semantic obviousness [25], and so on. Such promising progress offers us a wide variety of models to address the IQA problem. But meanwhile, it also becomes a sweet nuisance to fairly evaluate and compare them, which is crucial to guide the choice of various quality driven applications in the real world.

At present, the most common evaluation indicators for IQA are two classic rank correlation statistics, i.e., the Spearman’s $\rho$ and Kendall’s $\tau$ [26]. It is not surprising because the mean opinion score (MOS) collected from human subjects is typically represented by an ordinal variate in terms of rating scale [27, 28]. Computing the rank correlation between IQA prediction and MOS is a straightforward way to measure its capability of sorting out perceptually preferred image. However, there is little research discussing the suitability of directly applying Spearman’s and Kendall’s statistics to the IQA task, even they are successful in many other fields, such as, medicine [29], biology [30], neuroscience [31], and so on.

In reviewing these classic indicators [26], there are two rarely mentioned but noteworthy features, i.e., the uniform weighting and hard ranking principles, which assume the sorting mistakes in different quality levels sharing the same significance and each pair of images are sortable. Under these hypotheses, two important perceptual properties are neglected for the IQA task, which easily results a mismatch between the rank correlation evaluation and human preference. Firstly, in many real world applications, the sorting accuracy of high quality images usually plays a more important role than the poor quality ones, where only the top-ranked images are pushed to the users. To illustrate this feature, in Fig. 1, we compare two rank results of several denoised images. In this example, there is only one pair of images are mistakenly sorted for both the second and third rows, which are considered to share the same sorting accuracy in terms of the Spearman’s Rank Correlation Coefficient (SRCC) and Kendall’s Rank Correlation Coefficient (KRCC). However, as shown in Fig. 1, the recommended denoising result from the third row clearly outperforms the one in the second row. When the sorting error occurs in the top-rank, a perceptually poor image is more likely to be pushed to the user as shown in the second row. By contrast, the bottom-rank error for the poor quality images usually shows negligible impact on the recommending results, where the third row still outputs the perceptually best denoising result. Secondly, due to the inherent uncertainty in making judgement [32, 33], the ranking results between two images could not provide reliable measurement for the human perception of image quality, when they share the similar quality levels. For clarity, in Fig. 2, we compare two compressed images whose DMOSs are 13.72 and 13.70, respectively. Let $f(\cdot)$ and $g(\cdot)$ denote two IQA metrics towards DMOS, and we have the evaluation results of f[\text{Fig. \ref{fig_img_ref} (a)}]>f[\text{Fig. \ref{fig_img_ref} (b)}], g[\text{Fig. \ref{fig_img_ref} (a)}]<g[\text{Fig. \ref{fig_img_ref} (b)}]. When we follow the hard ranking principle in SRCC and KRCC, it is easy to conclude that $f(\cdot)$ is better than another one $g(\cdot)$. But, in terms of human perception, it is not certain for us to judge whose prediction is better between $f(\cdot)$ and $g(\cdot)$. Particularly, we implement a subjective investigation across 30 human subjects, who are compulsorily required to give a binary response, i.e., Fig. 2 (a) is better or worse than Fig. 2 (b) in terms of image quality. Unsurprisingly, the divisions are serious across all subjects, where 16 votes prefer Fig. 2 (a) and the others disagree with them. The only agreement between all subjects is that it is too hard to make a certain judgement in distinguishing Figs. 2 (a) and (b), which means the hard ranks between perceptually similar images do not contribute to fairly compare different IQA metrics.

Although the two problems mentioned above are widespread in the field of IQA, there are very little researches discussing an alternative rank correlation indicator other than SRCC and KRCC. In [34, 35], Wang et al. proposed an ingenious maximum differentiation (MAD) solution for comparing different IQA metrics, where each pair of IQA metrics fight with each other by detecting the most noticeable outliers from its competitors. In the following, two new indicators including the aggressiveness and resistance matrixes are designed for the group MAD (gMAD) [35], which compute the average subjective scores collected from pair comparison. It is worth nothing that the gMAD is initially designed to accelerate the comparison of different IQA algorithms across a large-scale test set. The perceptual importance variance of different quality levels and the subjective uncertainty are not fully considered by the aggressiveness and resistance indicators as well. In addition, the gMAD methodology requires additional manual work for rating the difference of pairwise images in evaluating a new IQA metric, which fairly limits the usability of the aggressiveness and resistance indicators.

To address the aforementioned challenges for IQA metric evaluation, we propose an easy-to-use perceptually weighted rank correlation (PWRC) indicator in this paper. In comparison with the popular SRCC and KRCC, the proposed indicator is characterized by three new features which are summarized in the following:

Nonuniform weighting: To capture the perceptual importance variance in computing the rank correlation, we assign different weights to the rank mistakes occurring in different quality levels. For a pair of images, the weight is jointly determined by the maximum rank of their MOS/DMOS and the difference between their MOS/DMOS. That is, we focus on penalizing great rank mistakes in the high quality levels. 2. 2.

Selective activation: In view of the subjective uncertainty for comparing perceptually similar images, we only count the perceptible sorting mistakes for the pairwise images whose MOS/DMOS difference exceeds a given sensory threshold (ST). In this way, the proposed indicator could suppress the impact of insensitive sorting errors in measuring the rank correlation between IQA predictions and human-rated MOS/DMOS. 3. 3.

Graphical representation: Unlike the traditional numerical representation that shows rank performance with single value, we utilize a graphical plot to illustrate the sorting accuracy (SA) variation with a changeable ST, which is named by SA-ST curve. The graphical representation could compare different IQA metrics more intuitively. Meanwhile, the SA-ST curve is capable to measure the superiority of an IQA metric under different confidence intervals, which is beyond both SRCC and KRCC.

Extensive experiments on five publicly available databases show that the proposed PWRC indicator could better distinguish different IQA metrics even they share the same number of mistakenly ranked image pairs. Furthermore, we also validate the applicability of the proposed indicator in recommending robust IQA metrics for both the degraded and enhanced images.

The remainder of this paper is organized as follows. Section II reviews the limitations of the classic SRCC and KRCC statistics. Then, the proposed PWRC indicator is introduced in Section III. We discuss the experimental results in Section IV. Finally, this paper is concluded in Section V.

II Rank Correlation Measurement Based on equal weighting

Given $n$ objects considered referring to two variables $\textbf{x}=\{x_{i}\}_{i\leq n}$ and $\textbf{y}=\{y_{i}\}_{i\leq n}$, Kendall et al. [26] designed a generalized correlation coefficient $\Gamma$ which is given by

[TABLE]

where $a_{ij}$ denotes the $x$-correlation variable between $x_{i}$ and $x_{j}$, and $b_{ij}$ denotes the $y$-correlation variable between $y_{i}$ and $y_{j}$. Both the $a_{ij}$ and $b_{ij}$ are required to be anti-symmetric and satisfy $a_{ij}=-a_{ji}$, $b_{ij}=-b_{ji}$.

By properly setting $a_{ij}$ and $b_{ij}$, we can easily deduce the Spearman’s $\rho$ and Kendall’s $\tau$ from Eq. (1). Let $a_{ij}$ and $b_{ij}$ represent the rank deviation of a pair of observation values

[TABLE]

where $p_{i}$ denotes the rank of $x_{i}$ in x, and $q_{i}$ denotes the rank of $y_{i}$ in y. Then, we can obtain $\rho$ as

[TABLE]

Assuming there are no tied ranks in x and y, the rank variables $\{p_{i}\}_{i\leq n}$ and $\{q_{i}\}_{i\leq n}$ would share equally distributed values ranging in $[1,n]$. Under this assumption, it is easy to deform Eq. (3) to a more familiar definition for Spearman’s $\rho$ according to [26], i.e.,

[TABLE]

where $d_{i}=i-q_{i}$, $w=6/(n^{3}-n)$, and $p_{i}$ denotes the ground-truth ranks represented by a sequential value $p_{i}=i$.

It is clear that Spearman’s $\rho$ assigns the same weight $w$ to each rank mistake term $d_{i}^{2}$. It could reflect the subjective uncertainty in some extent, where an insensitive rank mistake results a smaller $d_{i}^{2}$. But, the perceptual importance variance is completely ignored by its uniform weight $w$.

Similarly, when we set $a_{ij}$ and $b_{ij}$ to the binary rank discriminator, i.e.,

[TABLE]

we can deduce the Kendall’s $\tau$ as

[TABLE]

where $w=2/(n^{2}-n)$, and $\text{sgn}(\cdot)$ is a binary signum function defined by

[TABLE]

It is seen that the Kendall’s $\tau$ works like a outlier detector, which tries to capture the disagreement of the rank-orders between $(p_{i},p_{j})$ and $(q_{i},q_{j})$. Because Eq. (6) only counts the signs of pairwise ranks, it is unable to measure the perceptibility of this outlier, which is determined by the severity of sorting error. Meanwhile, the uniform weight $w$ is also applied to each pair of ranked samples in Eq. (6), which does not consider the perceptual importance variance across different rank levels.

III Perceptually Weighted Rank Correlation Indicator

To better interpret the visual perception behaviors in measuring rank correlation, we propose to assign different weights to each pair of ranked samples by considering both the sorting error degree and the level of the mistaken ranks. Meanwhile, the perceptibility of the mistaken ranks are also identified via a mutable sensory threshold.

Suppose $\textbf{x}=\{x\}_{i\leq n}$ and $\textbf{y}=\{y\}_{i\leq n}$ are the continuous-valued MOSs/DMOSs and IQA predictions, respectively. We first transform them to the discrete rank-orders $\textbf{p}=\{p\}_{i\leq n}$ and $\textbf{q}=\{q\}_{i\leq n}$. Then, the proposed indicator separates the task of rank correlation measurement into three steps including comparison activation, outlier detection, and importance measurement.

Before comparing the agreement between each pair of ($x_{i}$, $x_{j}$) and ($y_{i}$, $y_{j}$), we need to judge wether this comparison is perceptually sensitive or not. When the MOSs/DMOSs difference between $x_{i}$ and $x_{j}$ exceeds the sensory threshold, we will activate this comparison between ($x_{i}$, $x_{j}$) and ($y_{i}$, $y_{j}$). As discussed in [36, 37, 38, 39, 40], the visual sensitivity covers a wide dynamic range across different ages, genders, consciousness, and so on. So, we sample multiple sensory thresholds $\textbf{t}=\{t\}_{k\leq K}$ to evaluate the rank correlation, where the activation function is denoted by $A(\textbf{x},T)$ and $T=t_{k}$. In the following, the outlier detection focus on identifying discordant rank-orders between ($p_{i}$, $p_{j}$) and ($q_{i}$, $q_{j}$), and we denoted it by $D(\textbf{p},\textbf{q})$. Given all of the activated mistaken rank-orders, we need to assign different weights to them by measuring their perceptual importance. Both the rank-order deviation of (p, q) and the location of the mistaken rank are considered in the importance measurement, which is denoted by $M(\textbf{p},\textbf{q})$. Finally, by combining these three components together, we can yield an overall sorting accuracy indicator, i.e.,

[TABLE]

where $f(\cdot)$ is a combination function for fusing three rank correlation related components.

In defining the rank correlation indicator $S(\textbf{x},\textbf{y},T)$, we would like it to possess both the usability and visual perception features, which should satisfy four attributes.

Symmetry: $S(\textbf{x},\textbf{y},T)=S(\textbf{y},\textbf{x},T)$; 2. 2.

Boundness: $S(\textbf{x},\textbf{y},T)\leq 1$; 3. 3.

Ambiguity: Given a pair of inputs x and y, their rank correlation is non-unique and varies with a changeable sensory threshold $T$; 4. 4.

Unique maximum: $S(\textbf{x},\textbf{y},T)=1$, if and only if x = y and all elements of x satisfy $\|x_{i}-x_{j}\|\gg T$ when $i\neq j$.

At first, we define the activation function as

[TABLE]

where the constant $C_{1}$ is used to control the steepness of the activation curve. When $C_{1}$ grows larger, the activation function would get close to the unit step function. In comparison with the hard activation by unit step function, the proposed soft activation in Eq. (9) could better model the progressive change of visual sensitivity [41, 42].

Following the Gaussian distribution assumption about human opinion score [43], we consider the MOSs/DMOSs variables satisfy $\textbf{x}\sim\mathcal{N}(x_{i},\sigma_{i})$ and deduce an appropriate $C_{1}$ via interval estimation, where $x_{i}$ and $\sigma_{i}$ are the mean value and standard deviation of human opinion scores for the $i$th image. Based on the three-sigma rule [44], we know that there are about 95% human opinion scores lying within two standard deviations away from a MOS/DMOS in evaluating each image. In Fig. 3, we show the distribution of human opinion scores, which is modeled by a Gaussian distribution. The sensory threshold $T$ adjusts the mean value of this Gaussian distribution to $x_{i}-T$. When another MOS/DMOS $x_{j}$ goes beyond $x_{i}-T$ by the interval of $2\sigma_{i}$, we believe that these two images are perceived differently at the confidence level of 0.95, which results the equation

[TABLE]

where $C_{1}$ is approximately equal to $3/(2\sigma_{i})$. It is noted that different IQA databases usually collect the MOS/DMOS $x_{i}$ and standard deviation $\sigma_{i}$ at different scales. To obtain a general setting for $C_{1}$, we normalize all subjective scores to the range of [0, 100]. Let $\omega=1/(\max{\{\textbf{x}\}}-\min{\{\textbf{x}\}})$ denote the scaling factor and $\varepsilon=-\min{\{\textbf{x}\}}/(\max{\{\textbf{x}\}}-\min{\{\textbf{x}\}})$ denote the bias term for x. Then, the normalization for x is defined by

[TABLE]

Meanwhile, all standard deviations $\bm{\sigma}=\{\sigma\}_{i\leq n}$ are normalized by

[TABLE]

In this paper, the mean value of $\hat{\bm{\sigma}}$ across three popular IQA databases, i.e., LIVE II [8], TID2013 [9] and ChallengeDB [11], is utilized to compute $C_{1}$, where $E(\hat{\bm{\sigma}})=8.577$ and $C_{1}=3/(2\times 8.577)=0.175$.

The outlier detection function borrows the idea of sign comparison from Kendall’s $\tau$

[TABLE]

where each pair of mistaken rank-orders is labeled by -1. Otherwise, the outlier detection function would denote them by 1. Clearly, more discordant pairs between ($p_{i}$, $p_{j}$) and ($q_{i}$, $q_{j}$) would lead more negative outputs from Eq. (13).

In comparing each pair of rank-orders ($p_{i}$, $p_{j}$) and ($q_{i}$, $q_{j}$), we assign different weights to them by measuring their perceptual importance. Two factors, i.e., rank deviation and rank level, are considered for computing the weight. Particularly, the ground-truth label $p_{i}$ has been sequentially ranked as $\{p_{i}=i\}_{i\leq n}$. We represent the normalized rank deviation term between $(p_{i},p_{j})$ and $(q_{i},q_{j})$ by $d_{ij}=(\|i-q_{i}\|+\|j-q_{j}\|)/(2n-2)$. The normalized rank level term is $l_{ij}=(\max\{i,j\}-1)/(n-1)$. Both $d_{ij}$ and $l_{ij}$ locate in the range [0, 1]. Then, the importance measurement function is defined as

[TABLE]

where $M(\textbf{p},\textbf{q})\in[0,1]$ when $n>1$, and $\sum_{i\neq j=1}^{n}M(\textbf{p},\textbf{q})=1$. Particularly, a larger weight would be assigned to the predicted rank-order $q_{i}$ which deviates more from a high level ground-truth rank $p_{i}$. For clarity, the weight variation across different rank deviations and rank levels is shown in Fig. 4.

Finally, we combine all three components in Eqs. (9), (13) and (14) to compute the proposed PWRC indicator, i.e.,

[TABLE]

It is clear that the Eq. (15) satisfy all four attributes for designing easy-to-use and perception aware rank correlation indicator. Particularly, the Kendall’s $\tau$ corresponds to a special case of Eq. (15) when we set the comparison activation and importance measurement functions to constants, i.e., $A(\textbf{x},T)=1$ and $M(\textbf{p},\textbf{q})=2/(n^{2}-n)$.

To highlight the difference between PWRC and existing rank correlation indicators, we investigate the evaluation accuracy of different indicators on a set of synthetic data. Assuming that there are five MOS values, i.e., $n=5$, $\textbf{x}=\{x_{1}=5,x_{2}=10,x_{3}=20,x_{4}=35,x_{5}=55\}$, and the rank of each MOS is given by $\textbf{p}=\{p_{1}=1,p_{2}=2,p_{3}=3,p_{4}=4,p_{5}=5\}$. For comparison with the single-valued indicators SRCC and KRCC, the activation function in PWRC is first set to a constant, i.e., $A(\textbf{x},T)=1$. In this investigation, we set the ground-truth rank of x to a descending order permutation $p_{5}\sim p_{1}$, which is widely used in kinds of image retrieval and enhancement applications. Then, ten predicted ranks indexed by $\text{S}1\sim\text{S}10$ are evaluated based on different indicators, where the detail results are shown in Table I. For analysis, the total number of mistaken rank-orders, i.e., $L=\Sigma_{i\neq j=1}^{n}w[D(\textbf{p},\textbf{q})]$, is also calculated for each predicted rank, where $w(\cdot)$ is a window function and defined by

[TABLE]

It is seen that all rank correlation indicators gradually reduce when $L$ increases from 0 to 10. If $L$ is fixed, the SRCC and KRCC would keep the same even the predicted ranks are different. By contrast, the proposed PWRC could further tell the difference between different predicted ranks, which possess the same $L$. To verify the significance of this subtle discriminability, we analyze the average MOS difference between the top N ranked samples and the rest ones, which is computed by

[TABLE]

where a larger $\triangle d_{N}$ means the predicted rank is better for recommending high quality images. Since $N$ typically varies across different applications, we further calculate the mean value of $\triangle d_{N}$ by

[TABLE]

As shown in Table I, $\triangle\text{MOS}$ monotonically decreases from the predicted rank S1 to S10. Only the proposed PWRC captures this change, which is crucial for recommending reliable IQA metrics to various quality driven applications.

After the comparison between single-valued indicators, we further illustrate the ambiguity of the proposed PWRC by resorting $A(\textbf{x},T)$ to Eq. (9). Particularly, twenty evenly spaced sensory thresholds $T$ ranging in [0, 100] are investigated in this section, which produce a SA-ST curve for each predicted rank. As shown in Fig. 5, the PWRC indicator dynamically changes when $T$ increases from 0 to 100. Particularly, there are two important features of PWRC revealed from this observation.

Firstly, the PWRC is non-monotonic with respect to an increasing threshold $T$, which may result in different evaluation results in comparing two predicted ranks. For example, in Fig. 5, the predicted rank S4 is superior to S5 when $T$ is smaller than 10. As $T$ increases from 15 to 30, S4 becomes inferior to S5. When $T$ is larger than 35, their performances become close to each other. The similar observations could also be found between S7 and S8. The main reasons behind this observation lie in the contrary effects of the inactivation $A(\textbf{x},T)$ towards different responses in $D(\textbf{p},\textbf{q})$ and the nonuniform weighting $M(\textbf{p},\textbf{q})$ for different quality levels. More specifically, given two pairs of correctly and mistakenly ranked images, their inactivations would bring PWRC reduction and elevation, respectively. In addition, when the inactivation is applied to high quality image pairs, the PWRC reduction or elevation would be more significant. Otherwise, the PWRC change would be relatively small. This feature enables PWRC to evaluate an IQA metric more comprehensively, where a changeable threshold $T$ could capture the visual sensitivities across different genders, ages and so on [36, 37].

Secondly, although the PWRC shows different variation tendencies for various predicted ranks, they tend to approach each other with an increasing $T$. This feature enhances the fault-tolerant capability of PWRC with respect to unsortable images, whose MOSs/DMOSs are close to each other. More specifically, if an IQA metric’s PWRC outperforms the others on a larger $T$, the confidence of its superiority would be higher. By accumulating PWRC across a given threshold range [$T_{min}$, $T_{max}$], we derive a confidence-aware rank correlation measurement from the area under the curve (AUC), which is defined by

[TABLE]

where $T_{min}=\min\{2\hat{\bm{\sigma}}\}$ and $T_{max}=\max\{2\hat{\bm{\sigma}}\}$. $\hat{\bm{\sigma}}$ is the set of all normalized standard deviations associated to the MOSs/DMOSs in each IQA database. For clarity, an illustration of $\text{AUC}_{ca}$ is shown in Fig. 6.

IV Experiments

IV-A Protocols

In this section, we investigate the performance of different rank correlation indicators on five publicly available IQA databases, which include the LIVE II [8], TID2013 [9], ChallengeDB [11], IVCDehazing [45] and ESPL-LIVE [46]. Particularly, each database explores the human perception of image quality towards different visual contents. The LIVE II and TID2013 databases collect the human opinion scores on thousands of images contaminated by artificially simulated distortions, such as, JPEG2000, JPEG, additive Gaussian white noise, Gaussian blur, fast fading and so on. In ChallengeDB, there are 1162 authentically distorted images collected from a wide variety of mobile camera devices including the smart phones and tablets. The IVCDehazing database focuses on studying the IQA problem for the enhanced image under foggy weather. For the ESPL-LIVE database, a large-scale subjective scores are collect on 1811 high dynamic range (HDR) images which are created by different tone mapping and multi-exposure fusion techniques.

For comparison, we use different rank correlation indicators to evaluate thirteen popular IQA algorithms, which include six full-reference (FR) metrics, i.e., PSNR [47], IWPSNR [48], SSIM [20], IWSSIM [12], MSSSIM [49], FSIM [50], and seven no-reference (NR) metrics, i.e., BIQI [51], BLIINDS II [52], BRISQUE [53], DIIVINE [54], NFERM [55], M3 [56] and TCLT [57].

Since the NR-IQA algorithms require additional training process, we follow the random split criteria in [51, 52, 54, 53, 55, 56, 57] and separate each IQA database into non-overlapped training and testing sets over 1000 times. More specifically, in each trial, we randomly choose part of reference images and their contaminated or enhanced versions to construct the training set. The testing set is composed of the rest images, whose visual contents are different from the training set. It is noted that all images collected in ChallengeDB database do not contain any reference images or overlapped visual contents. So, we straightforwardly divide it to two parts for training and testing, respectively. In addition, all FR-IQA algorithms are evaluated on the same test set like the other NR-IQA metrics, which ensures a fair comparison under consistent experimental setup.

In this section, three split ratios are investigated where the training set takes up 80, 50, and 20 percents of images in each database. The median values of SRCC, KRCC and $\text{AUC}_{ca}$ across 1000 random split trials are used to rank different IQA algorithms. Since the quality-driven applications prefer an accurate push capability for high quality images, we employ $\Delta\text{MOS}$ as the benchmark for ranking different IQA metrics. Given a rank correlation indicator, the disagreements between its rank results and $\Delta\text{MOS}$ are used for measuring the push accuracy, where more disagreements mean worse accuracy.

IV-B Implementation Details

As discussed in Section III, we compute the proposed PWRC on normalized MOS/DMOS values and standard deviations. Given an IQA database, the scaling factor $\omega$ and bias term $\varepsilon$ are computed on the set of all MOS/DMOS values. Both the ground-truth subjective scores and the predicted image qualities are normalized by Eqs. (11) and (12). Then, we compute the confidence-aware indicator $\text{AUC}_{ca}$ from PWRC curve to rank different IQA metrics, where a sensory threshold range [$T_{min}$, $T_{max}$] is required as shown in Eq. (19). It is noted that different IQA databases would produce different $\omega$, $\varepsilon$ and [$T_{min}$, $T_{max}$]. For clarity, we summarize the parameters of different databases in Table II, where a larger MOS/DMOS standard deviation would require wider sensory threshold range for computing $\text{AUC}_{ca}$.

IV-C Evaluation on Degraded Images

In this section, we first employ the proposed PWRC to evaluate the rank performance of different IQA metrics towards degraded images, which are investigated on LIVE II, TID2013 and ChallengeDB databases, respectively. To illustrate an intuitive comparison results, the SA-ST curves of different IQA algorithms are shown on Fig. 7. It is noted that the ChallengeDB database does not contain uncontaminated reference images. So only the results of NR-IQA algorithms are reported in Fig. 7. It is seen that existing IQA metrics work well on artificially simulated distortions, where most PWRC results are larger than 0.5 on LIVE II and TID2013 databases. However, in coping with the authentically distorted images, the performances of all IQA metrics are very poor, whose PWRC results are all smaller than 0.4 on the ChallengeDB database. In addition, it is found that the performances of the training based NR-IQA algorithms would gradually decrease when the ratio of training set reduces from 80% to 20%. By contrast, the FR-IQA metrics achieve more robust results, where the similar results could also be found in [35]. As shown in Fig. 7 (a), all NR-IQA algorithms, apart from BIQI, achieve comparable PWRC performance in comparison with the FR-IQA metrics when the training set takes up 80% images. While, in Fig. 7 (g), the FR-IQA algorithms outperform most NR-IQA algorithms when the training set ratio drops to 20%.

To quantitatively compare the rank results towards different IQA metrics, we further report their SRCC, KRCC, $\text{AUC}_{ca}$ and $\Delta\text{MOS}$ values on LIVE II, TID2013 and ChallengeDB databases, which are shown in Tables III, IV and V. Particularly, we highlight all pairwise metrics by boldface and italic when they present inconsistent rank results between $\Delta\text{MOS}$ and the other indicators. Each mistaken pair is labeled by the same color, such as, red, blue and green. As shown in Table III, when 80% images are used for training on the LIVE II database, the SRCC considers BLIINDS II and BRISQUE present the same rank accuracy, whose reported values are both 0.925. But, the $\Delta\text{MOS}$ value of BRISQUE is clearly better than BLIINDS II. Both the KRCC and the proposed $\text{AUC}_{ca}$ report the consistent rank result with respect to $\Delta\text{MOS}$. When IW-PSNR is compared with NFERM, the SRCC prefers the FR-IQA metric IW-PSNR, which is also opposite to the $\Delta\text{MOS}$ and the other indicators. When the training set ratio is set to 50%, both the SRCC and KRCC would recommend SSIM in comparison with M3. However, we can find that M3 achieves higher $\Delta\text{MOS}$, and the proposed $\text{AUC}_{ca}$ indicator correctly reflects this superiority. The similar result is also reported in comparing M3 with TCLT when we use 20% images for training. Meanwhile, more disagreements between $\Delta\text{MOS}$ and SRCC/KRCC could be found in Tables IV and V.

Across all of experiments on LIVE II, TID2013 and ChallengeDB databases, we can find that the $\text{AUC}_{ca}$ computed from our proposed PWRC achieves highly consistent rank results with respect to $\Delta\text{MOS}$, where there is no disagreement between them. It verifies that the proposed PWRC indicator is beneficial for pushing perceptually preferred images from the degraded data, where the top ranked images possess higher average MOS.

IV-D Evaluation on Enhanced Images

Reliable IQA metric plays a crucial role in kinds of quality driven applications. In this section, we utilize the proposed PWRC indicator to recommend appropriate IQA metrics for two image enhancement applications, i.e., image dehazing and HDR image reconstruction, which are investigated on IVCDehazing and ESPL-LIVE databases, respectively. More specifically, the IQA metric serves as a selector towards different enhancement algorithms. The PWRC aims to find the most robust selector from some candidate IQA metrics. It is worth nothing that there is no perfect reference for an enhanced image. So, we select seven state-of-the-art NR-IQA algorithms as the candidate metrics, which include BIQI [51], BLIINDS-II [52], BRISQUE [53], DIIVINE [54], NFERM [55], M3 [56] and TCLT [57].

For fair comparison, different enhancement algorithms should be applied to the same visual content. In view of this fact, we develop a slightly deformed average MOS difference to represent the benchmark of each IQA metric in pushing perceptually preferred images. Given $K$ candidate algorithms and $L$ raw images, we use $\Delta\text{MOS}_{k,l}$ to denote the performance of the $k$th IQA metric for ranking $K$ enhanced versions of the $l$th raw image, where $\Delta\text{MOS}_{k,l}$ could be computed via Eq. (18) by setting $n$ to $K$. Then, the overall performance of the $k$th IQA metric could be defined by

[TABLE]

where the similar measurement could be found in the rational test of [35].

Similar to $\Delta\overline{\text{MOS}}_{k}$, we use the image-wise averaged rank correlation indicators to sort different enhancement algorithms

[TABLE]

where the area under the curve values computed from $\overline{\text{PWRC}}$ is denoted by $\overline{\text{AUC}}_{ca}$. For comparison, the deformed indicators $\overline{\text{SRCC}}$ and $\overline{\text{KRCC}}$ are also tested in this section, which can be computed by replacing $\text{PWRC}_{k,l}$ with $\text{SRCC}_{k,l}$ and $\text{KRCC}_{k,l}$ in Eq. (21), respectively.

Similar to the test on degraded images, we first show the SA-ST curves of different IQA metrics in Fig. 8. It is seen that the existing NR-IQA metrics work poorly in recommending high quality dehazing or HDR image, whose $\overline{\text{PWRC}}$ values are all very small. In addition, in comparison with Fig. 7, we can clearly find more intersections between different SA-ST curves as shown in Fig. 8. For example, in Fig. 8 (a), the BRISQUE outperforms M3 when $T$ is smaller than 40. However, because the descent speed of BRISQUE is much higher than M3, its $\overline{\text{PWRC}}$ values become lower than M3 when $T$ is larger than 40. The similar observations could also be found between NFERM and BLIINDS II as shown in Fig. 8 (b). In addition, the intersection between BRISQUE and BLIINDS II also occurs in Figs. 8 (e) and (f), and so on. We can find that the intersection between different SA-ST curves occurs more frequently for the IQA metrics whose $\overline{\text{PWRC}}$ locate in low values as shown in Figs. 8 (a)-(c). This is because that there are two factors could result in a low PWRC value, i.e., 1) a large number of correct ranks locate in low quality levels, or 2) a small amount of correct ranks locate in high quality levels. In case 1), the descent speed of SA-ST curve would be relatively slow, where the changes in low quality levels only slightly impair $\overline{\text{PWRC}}$ due to their low weights. By contrast, in case 2), the descent speed of SA-ST curve would be faster due to the larger weights assigned to high quality levels in $\overline{\text{PWRC}}$. This subtle discriminability enables $\overline{\text{PWRC}}$ to evaluate each IQA metric from a more comprehensive perspective. In the following, by means of $\overline{\text{AUC}}_{ca}$, we can derive a single-valued measurement from $\overline{\text{PWRC}}$ under the given confidence intervals.

In Tables VI and VII, we report the detailed accuracy of different IQA metrics on the IVCDehazing and ESPL-LIVE databases, respectively. Similar to the observations in [45, 46], existing NR-IQA metrics work poorly in ranking the qualities of enhanced images. As shown in Table VI, many IQA metrics produce negative $\Delta\overline{\text{MOS}}$ values. The results in Table VII are slightly better, whose $\Delta\overline{\text{MOS}}$ values are still very small. All three rank correlation indicators could reflect the terrible accuracy of these algorithm selectors, where $\overline{\text{SRCC}}$, $\overline{\text{KRCC}}$ and $\overline{\text{AUC}}_{ca}$ are all very low. But, there are also disagreements between the detailed ranks of IQA metrics predicted by them, where the discordant IQA pairs are highlighted by boldface and italic in Tables VI and VII.

As shown in Table VI, when 80% dehazing images are used for training, both $\overline{\text{AUC}}_{ca}$ and $\overline{\text{KRCC}}$ recommend BRISQUE instead of M3 or TCLT, which is consistent with $\Delta\overline{\text{MOS}}$. While, the $\overline{\text{SRCC}}$ prefers TCLT. In Table VII, when we use 50% HDR images for training, both $\overline{\text{SRCC}}$ and $\overline{\text{KRCC}}$ consider BRISQUE is better than BLIINDS II. Only the proposed $\overline{\text{AUC}}_{ca}$ indicator is consistent with $\Delta\overline{\text{MOS}}$. The similar condition is also found for comparing BRISQUE with BLIINDS II, when the training set ratio is 20%. These experiments confirm that the proposed $\overline{\text{PWRC}}$ indicator works better for recommending robust IQA metric towards enhanced images, whose top ranked samples would present higher average perceptual qualities.

V Conclusion

In this paper, we propose a perceptually weighted rank correlation (PWRC) indicator to fairly compare different IQA algorithms. Inspired by two important visual perception properties, i.e., perceptual importance variation and subjective uncertainty, we develop the nonuniform weighting and adaptive activation schemes to evaluate the rank accuracy of each IQA algorithm. More specifically, a larger weight would be assigned to the image pair with higher quality level and greater rank deviation. Meanwhile, the comparison between two images would be activated only if their rank deviation exceeds the given sensory threshold, which suppresses the interference from perceptually unsortable image pairs. Extensive experiments on five publicly available IQA databases show that the proposed indicator is more consistent with human perception and works better for recommending perceptually preferred images.