ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images

TL;DR

This paper introduces ESIQAnet, a novel perceptual quality assessment model for egocentric spatial images in XR, supported by the first dedicated IQA database, ESIQAD, with extensive experimental validation.

Contribution

The paper presents the first IQA database for egocentric spatial images and a new multi-stage feature fusion model, ESIQAnet, for accurate quality prediction across different XR display modes.

Findings

ESIQAnet outperforms 22 state-of-the-art IQA models.

The ESIQAD database contains 500 images with MOS under three display modes.

The model effectively captures stereoscopic and egocentric image quality features.

Abstract

With the development of eXtended Reality (XR), photo capturing and display technology based on head-mounted displays (HMDs) have experienced significant advancements and gained considerable attention. Egocentric spatial images and videos are emerging as a compelling form of stereoscopic XR content. The assessment for the Quality of Experience (QoE) of XR content is important to ensure a high-quality viewing experience. Different from traditional 2D images, egocentric spatial images present challenges for perceptual quality assessment due to their special shooting, processing methods, and stereoscopic characteristics. However, the corresponding image quality assessment (IQA) research for egocentric spatial images is still lacking. In this paper, we establish the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first IQA database dedicated for egocentric spatial images as far as we know. Our ESIQAD includes 500 egocentric spatial images and the corresponding mean opinion scores (MOSs) under three display modes, including 2D display, 3D-window display, and 3D-immersive display. Based on our ESIQAD, we propose a novel mamba2-based multi-stage feature fusion model, termed ESIQAnet, which predicts the perceptual quality of egocentric spatial images under the three display modes. Specifically, we first extract features from multiple visual state space duality (VSSD) blocks, then apply cross attention to fuse binocular view information and use transposed attention to further refine the features. The multi-stage features are finally concatenated and fed into a quality regression network to predict the quality score. Extensive experimental results demonstrate that the ESIQAnet outperforms 22 state-of-the-art IQA models on the ESIQAD under all three display modes. The database and code are available at https://github.com/IntMeGroup/ESIQA.