Blind Omnidirectional Image Quality Assessment: Integrating Local Statistics and Global Semantics

TL;DR

This paper introduces S$^2$, a blind omnidirectional image quality assessment method that combines local statistical features and global semantic features to accurately predict perceptual quality.

Contribution

The paper presents a novel approach that integrates local statistics and global semantics for blind omnidirectional image quality assessment, improving prediction accuracy.

Findings

S$^2$ achieves competitive performance against state-of-the-art methods.

Combining local statistics and global semantics enhances quality prediction.

The method effectively predicts perceptual quality without reference images.

Abstract

Omnidirectional image quality assessment (OIQA) aims to predict the perceptual quality of omnidirectional images that cover the whole 180$\times$360$^{\circ}$ viewing range of the visual environment. Here we propose a blind/no-reference OIQA method named S$^2$ that bridges the gap between low-level statistics and high-level semantics of omnidirectional images. Specifically, statistic and semantic features are extracted in separate paths from multiple local viewports and the hallucinated global omnidirectional image, respectively. A quality regression along with a weighting process is then followed that maps the extracted quality-aware features to a perceptual quality prediction. Experimental results demonstrate that the proposed S$^2$ method offers highly competitive performance against state-of-the-art methods.

Tables

Table 1. Table 1 : Performance comparisons of objective models.

Types	Methods	SROCC	PLCC	RMSE
FR-IQA	PSNR	0.6239	0.7008	9.9599
	SSIM [23]	0.8842	0.9002	6.0793
	MS-SSIM [10]	0.8222	0.8521	7.3072
	FSIM [24]	0.9152	0.9340	4.9864
	DeepQA [25]	0.9292	0.9375	4.8574
FR-OIQA	S-PSNR [1]	0.6449	0.7083	9.8564
	WS-PSNR [2]	0.6107	0.6729	10.3283
	CPP-PSNR [3]	0.6265	0.6871	10.1448
NR-IQA	BRISQUE [26]	0.8180	0.8376	7.6271
	BMPRI [27]	0.7470	0.7919	8.5258
	DB-CNN [28]	0.9308	0.9356	4.9311
NR-OIQA	MFILGN [4]	0.9670	0.9751	3.1036
	MC360IQA [5]	0.9428	0.9429	4.6506
	VGCN [6]	0.9639	0.9651	3.6573
	AHGCN [7]	0.9623	0.9643	3.6990
	Proposed S2	0.9710	0.9781	2.8945

Table 2. Table 2 : Performance comparisons for different viewport numbers in the statistic path.

Numbers	SROCC	PLCC	RMSE
$6$	0.9684	0.9769	3.0083
$20$	0.9686	0.9777	2.9626
$80$	0.9683	0.9771	2.9501

Table 3. Table 3 : Performance comparisons for different neural network architectures in the semantic path.

Architectures	SROCC	PLCC	RMSE
VGG-F	0.9497	0.9537	4.2107
VGG-M	0.9517	0.9576	4.0329
VGG-S	0.9451	0.9486	4.4345

Equations

$$\small G_{n}^{i}(x,y)=\left\{\begin{array}[]{l}V_{n},i=1\\ \sum\limits_{u=-2}^{2}{\sum\limits_{v=-2}^{2}k(u,v)G_{n}^{i-1}(2x+u,2y+v)},i>1\end{array}\right.,$$

$${\hat{G}}_n^i(x,y)=4\sum _{u=-2}^2\sum _{v=-2}^{2}k(u,v)G_n^i\left(\frac{u+x}{2},\frac{v+y}{2}\right).$$

$$L_n^i=G_n^i-{\hat{G}}_n^{i+1}.$$

$$f_{st}=\left[f_{GP},f_{LP}\right].$$

$$f_{se}=F{C}_1(D).$$

$$Q_{\text{overall }}=w{Q}_{st}+(1-w)Q_{\text{se }},$$

Full text

**Index Terms— ** Omnidirectional image, blind image quality assessment, low-level statistics, high-level semantics

1 Introduction

The rapid recent advancement in virtual reality (VR) technologies makes it possible to create immersive multimedia quality-of-experience (QoE) for end-users. As a representative form of VR, omnidirectional content has increasingly emerged in our daily life. To evaluate and optimize the perceptual QoE of omnidirectional content, objective omnidirectional image quality assessment (OIQA) models play a critical roles in the development of modern VR systems.

In the literature, objective OIQA models have emerged that follow both full-reference (FR) and no-reference (NR) frameworks. FR-OIQA models assume full access to information of the reference image and are usually direct extensions of traditional FR methods developed for regular rectangular 2D image quality assessment (IQA). For example, based upon the peak signal-to-noise ratio (PSNR), Yu et al. [1] propose the spherical PSNR (S-PSNR) algorithm, where PSNR is calculated for uniformly distributed points on a sphere instead of projected rectangular image. In [2], the weighted-to-spherically uniform PSNR (WS-PSNR) method is presented, where a weighting map is created by considering the stretched degree. Zakharchenko et al. [3] propose the Craster parabolic projection PSNR (CPP-PSNR) approach, which maps the reference and distorted omnidirectional images on the Craster parabolic projection followed by PSNR computation.

NR-OIQA methods do not require access to the reference image and are more desirable in many application scenarios. Existing NR-OIQA approaches can generally be classified into two categories, depending on whether the conventional hand-crafted or learned deep features are employed for quality prediction. Multi-frequency information and local-global naturalness are applied to develop the MFILGN model [4]. More recent models employ deep convolutional neural networks (CNNs) or graph convolution networks (GCNs). These models demonstrate promising performance, including the multi-channel CNN for blind 360-degree image quality assessment (MC360IQA) [5], the viewport oriented graph convolution network (VGCN) [6], and its variant named adaptive hypergraph convolutional network (AHGCN) [7].

In a 360-degree viewing environment, e.g. using a head-mounted device, the observer is not able to visualize the whole omnidirectional content simultaneously, and thus an important step in the human subjective viewing experience is to establish or reconstruct a sense of the global semantics by browsing and integrating information from many viewports. During the course of image quality assessment, such global semantics are integrated with local observations on image fidelity, naturalness, and/or artifacts to produce an overall quality evaluation. Motivated by this observation, we propose a statistic and semantic oriented quality prediction framework named S2 for blind OIQA as illustrated in Fig. 1, by integrating features extracted from both low-level image statistics of multiple local viewports and high-level semantics of the hallucinated global omnidirectional image. A quality regression module is then leveraged to map the collection of the quality-sensitive features extracted from the two separate paths to an overall prediction of the subjective quality rating. Extensive experimental results demonstrate that the proposed method is superior to many state-of-the-art quality assessment models. In addition, we make some interesting observations on the relationship between semantic confidence and image distortions, as well as how the individual components affect the ultimate quality prediction performance in ablation studies.

2 Proposed Method

The overall framework of the proposed S2 method is shown in Fig. 2, which consists of a statistic path, a semantic path, and a final quality regression step.

Since a variety of viewports are browsed by the viewers, we first convert the distorted omnidirectional image (OI) to multiple viewports. Given each input distorted OI denoted by $D$, we exploit the non-uniform viewport sampling strategy [8, 9] and obtain $N$ viewports $V_{n},n=1,2,...,N$.

To capture the multi-scale characteristics of the human visual system [10], we construct pyramid representations [11, 12] of multiple local viewports. Specifically, multi-level Laplacian pyramids [13] are created by iterative Gaussian filtering, down-sampling, and subtracting, resulting in Gaussian and Laplacian pyramids in the same process. For a specific viewport $V_{n}$, layers of the Gaussian pyramid are calculated as follows:

[TABLE]

where $i$ is the layer index of the Gaussian pyramid, $x\in[0,X)$ and $y\in[0,Y)$ are the pixel position indices in which $X$ and $Y$ are the image dimensions, and $k(u,v)$ denotes the generating kernel that is typically defined by the coefficients of a low pass filter such as a 2D Gaussian filter.

We then interpolate each layer of the Gaussian pyramid by:

[TABLE]

The residual between the current layer of the Gaussian pyramid and the interpolation result from the next layer defines the current layer of the Laplacian pyramid:

[TABLE]

Since the computation of the $i$-th layer in the Laplacian pyramid requires the $(i+1)$-th layer of the Gaussian pyramid, the number of layers in the Laplacian pyramid is one less than that in the Gaussian pyramid.

To extract features from the Gaussian pyramid, we compute the default uniform local binary pattern (LBP) descriptors, resulting in 59 statistics for each Gaussian layer. When a 3-layer Gaussian pyramid is employed, this leads to 177 Gaussian pyramid features denoted by $f_{GP}$. For a Laplacian pyramid, motivated by the success of natural scene statistics (NSS) in IQA research [14, 15, 16], we extract mean subtracted and contrast normalized coefficients, leading to 36 features for each layer. When a 2-layer Laplacian pyramid is employed, this results in 72 Laplacian pyramid features denoted by $f_{LP}$. The full statistic feature set $f_{st}$, one for each viewport, is obtained by concatenating the statistical features extracted from the Gaussian and Laplacian pyramids as:

[TABLE]

We employ the VGGNet trained on the large ImageNet dataset [17] as the semantic feature extraction backbone, mainly for its simplicity and ability to capture image distortion-related representations [18]. In [19], three different structures of VGGNet have been proposed to balance between complexity and accuracy, namely fast VGG (VGG-F), medium VGG (VGG-M), and slow VGG (VGG-S). Each of them contains 5 convolutional (Conv) layers and 3 fully connected (FC) layers. The first two FC layers have 4,096 neurons, while the last one has 1,000 nodes indicating the 1,000 classes for image recognition. In our current implementation, we select the deep features from the first FC layer of VGG-M as our semantic feature set $f_{se}$:

[TABLE]

To learn the mapping from features to quality labels, we feed the statistic features and semantic features separately to support vector regression (SVR) models [20], and denote the regressed statistic and semantic quality scores as $Q_{st}$ and $Q_{se}$, respectively. The overall quality score is calculated by a weighted average:

[TABLE]

where $w$ is a weighting factor that determines the relative importance of the statistic and semantic feature predictors.

3 Validation

3.1 Experimental Setup and Performance Comparison

We evaluate the proposed approach on the CVIQD subjective database [21], which is so far a relatively large and widely adopted database containing both omnidirectional images and their corresponding quality labels given by human subjects. It consists of 16 original images and 528 distorted images produced by three classic image or video coding technologies, namely JPEG, AVC, and HEVC. The subjective quality ratings in the form of mean opinion score (MOS) are rescaled to the range of [0, 100], for which a higher MOS represents better perceptual image quality.

To compare the performance of various IQA models, we take Spearman Rank-Order Correlation Coefficient (SROCC), Pearson Linear Correlation Coefficient (PLCC) and Root Mean Squared Error (RMSE) as the evaluation criteria. Before calculating the PLCC and RMSE, a 5-parameter logistic nonlinear fitting approach [22] is implemented to map the predicted quality into the subjective quality space.

The database is randomly divided into 80% data for training and the remaining 20% data for cross-validation. In order to relieve the uncertainty in training/testing splitting, we repeat this random-splitting and cross-validation process 100 times and report the median performance.

The performance of the proposed algorithm is compared against state-of-the-art quality assessment models, including five FR-IQA, three FR-OIQA, three NR-IQA, and four NR-OIQA methods. The results are shown in TABLE 1, where we observe that for FR-IQA metrics, the PSNR-based models are inferior to more advanced approaches such as structural (SSIM, MS-SSIM, FSIM) and deep learning (DeepQA) models. Somewhat surprisingly, the FR-OIQA methods do not help further improve upon FR-IQA approaches. By contrast, the NR-OIQA models show significant superiority over NR-IQA methods. This is likely due to their specific design to capture the characteristics of omnidirectional images. Among all metrics tested, the proposed S2 method demonstrates highly competitive performance.

3.2 Semantic Confidence Versus Image Distortion

Since the proposed method contains a semantic path, it is interesting to observe the relationship between semantic confidence and image distortion. An example of distorted omnidirectional images with different JPEG, AVC and HEVC distortion levels is shown in Fig. 3, where from the first column to the third column, we observe that as the degree of distortion increases, the semantic confidence level decreases. This suggests that semantic information may be highly related to perceptual image quality. It is also interesting to see that the semantic confidence shows various sensitivities to different distortion types. In particular, the drop in semantic confidence levels is much less in more advanced image/video coding method HEVC than in earlier JPEG and AVC encoders.

3.3 Ablation and Parameter Sensitivity Tests

We evaluate the contributions from the statistic and semantic paths by ablation experiments, and the results are shown in Fig. 4, where GP1, GP2 and GP3, respectively, represent the cases of using the first, second, and third layers of Gaussian pyramid statistics only. GP denotes the case of using three layers of Gaussian pyramid statistics. We find that the performance increases gradually. Similarly, LP1 and LP2, respectively, correspond to the cases of using the first and second layers of Laplacian pyramid statistics only, while LP denotes the case of using 2-layer Laplacian pyramid statistics. The results show that LP produces the best performance among the three. The cases of adopting the statistic path and the semantic path only are denoted by St and Se, respectively. It is observed that either path alone can achieve promising quality prediction performance, but adopting both paths (i.e. the All case) delivers the best performance. Relative speaking, the more dominant factor seems to be the statistic path. This may not be surprising as the statistic features come from different viewports directly visualized by human subject while the global semantics offer complementary information for additional cues in quality assessment.

Because different parameter settings may be employed in the implementations of the proposed framework, here we test the sensitivity of our model with regard to various viewport numbers and semantic architectures. The results are reported in TABLE 2 and TABLE 3, respectively. We can see that the proposed model is insensitive to the viewport number. This allows us to reduce the number of viewports (for example, 6) to alleviate the computational complexity in real-world applications. The results also show that VGG-M outperforms the other neural network architectures in the semantic path. The possible reason may be that VGG-M achieves a preferable tradeoff between algorithm complexity and accuracy, making it a desired option for deep semantic backbone.

4 Conclusion

We propose a novel S2 framework for blind omnidirectional image quality assessment that integrates both local low-level statistic and global high-level semantic features. Extensive experiments show that the proposed method achieves state-of-the-art performance. Observations on the relationship between semantic confidence and image distortion, and the ablation/sensitivity tests offer additonal useful insights. Under the same framework, more advanced models for statistic and semantic analysis may be employed in the future, aiming for more accurate QoE assessment models that may help drive the advancement of immersive multimedia systems.