View-aware Salient Object Detection for 360{\deg} Omnidirectional Image

TL;DR

This paper introduces a large-scale 360-degree omnidirectional image dataset for salient object detection, and proposes a view-aware detection method that effectively handles panoramic distortions, discontinuities, and scale variations.

Contribution

The paper creates the largest 360-degree ISOD dataset with pixel-wise annotations and proposes a novel view-aware detection method using a Sample Adaptive View Transformer.

Findings

The dataset is the largest and most challenging for 360-degree ISOD.

The proposed method outperforms 20 state-of-the-art ISOD techniques.

Experiments demonstrate the effectiveness of view-aware features in panoramic scenarios.

Abstract

Image-based salient object detection (ISOD) in 360{\deg} scenarios is significant for understanding and applying panoramic information. However, research on 360{\deg} ISOD has not been widely explored due to the lack of large, complex, high-resolution, and well-labeled datasets. Towards this end, we construct a large scale 360{\deg} ISOD dataset with object-level pixel-wise annotation on equirectangular projection (ERP), which contains rich panoramic scenes with not less than 2K resolution and is the largest dataset for 360{\deg} ISOD by far to our best knowledge. By observing the data, we find current methods face three significant challenges in panoramic scenarios: diverse distortion degrees, discontinuous edge effects and changeable object scales. Inspired by humans' observing process, we propose a view-aware salient object detection method based on a Sample Adaptive View Transformer (SAVT) module with two sub-modules to mitigate these issues. Specifically, the sub-module View Transformer (VT) contains three transform branches based on different kinds of transformations to learn various features under different views and heighten the model's feature toleration of distortion, edge effects and object scales. Moreover, the sub-module Sample Adaptive Fusion (SAF) is to adjust the weights of different transform branches based on various sample features and make transformed enhanced features fuse more appropriately. The benchmark results of 20 state-of-the-art ISOD methods reveal the constructed dataset is very challenging. Moreover, exhaustive experiments verify the proposed approach is practical and outperforms the state-of-the-art methods.