Distortion-aware Transformer in 360{\deg} Salient Object Detection

TL;DR

This paper introduces DATFormer, a Transformer-based model designed to effectively handle distortions in 360-degree images for salient object detection, leveraging distortion-adaptive modules and a learnable relation matrix to improve accuracy.

Contribution

The paper proposes a novel distortion-aware Transformer model with adaptive modules and a learnable relation matrix specifically for 360-degree salient object detection.

Findings

Outperforms existing 2D SOD and 360 SOD methods

Effective handling of projection distortions in 360-degree images

Improved detection accuracy on public datasets

Abstract

With the emergence of VR and AR, 360{\deg} data attracts increasing attention from the computer vision and multimedia communities. Typically, 360{\deg} data is projected into 2D ERP (equirectangular projection) images for feature extraction. However, existing methods cannot handle the distortions that result from the projection, hindering the development of 360-data-based tasks. Therefore, in this paper, we propose a Transformer-based model called DATFormer to address the distortion problem. We tackle this issue from two perspectives. Firstly, we introduce two distortion-adaptive modules. The first is a Distortion Mapping Module, which guides the model to pre-adapt to distorted features globally. The second module is a Distortion-Adaptive Attention Block that reduces local distortions on multi-scale features. Secondly, to exploit the unique characteristics of 360{\deg} data, we present a learnable relation matrix and use it as part of the positional embedding to further improve performance. Extensive experiments are conducted on three public datasets, and the results show that our model outperforms existing 2D SOD (salient object detection) and 360 SOD methods.