Panoramic Vision Transformer for Saliency Detection in 360{\deg} Videos

TL;DR

The paper introduces PAVER, a novel panoramic vision transformer framework that effectively detects saliency in 360° videos by handling distortions and discontinuities, outperforming existing models without supervision.

Contribution

PAVER leverages a Vision Transformer with deformable convolution for efficient, pretrained model integration and geometric approximation, advancing 360° video saliency detection.

Findings

Outperforms state-of-the-art on Wild360 benchmark

Operates without supervision or auxiliary data

Improves omnidirectional video quality assessment results

Abstract

360$^\circ$ video saliency detection is one of the challenging benchmarks for 360$^\circ$ video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360$^\circ$ videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature. We present a new framework named Panoramic Vision Transformer (PAVER). We design the encoder using Vision Transformer with deformable convolution, which enables us not only to plug pretrained models from normal videos into our architecture without additional modules or finetuning but also to perform geometric approximation only once, unlike previous deep CNN-based approaches. Thanks to its powerful encoder, PAVER can learn the saliency from three simple relative relations among local patch features, outperforming state-of-the-art models for the Wild360 benchmark by large margins without supervision or auxiliary information like class activation. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision, including head movement.