360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos

TL;DR

This paper introduces a novel representation and a comprehensive dataset for visual object tracking and segmentation in 360-degree videos, addressing challenges posed by spherical distortion and wide field-of-view.

Contribution

It proposes the extended bounding field-of-view (eBFoV) representation and a new 360VOS dataset, advancing the development and evaluation of omnidirectional tracking and segmentation algorithms.

Findings

The proposed framework outperforms existing methods on the new dataset.

The 360VOS dataset includes 290 annotated sequences across diverse categories.

Tailored evaluation metrics enable rigorous assessment of omnidirectional tracking and segmentation.

Abstract

Visual object tracking and segmentation in omnidirectional videos are challenging due to the wide field-of-view and large spherical distortion brought by 360{\deg} images. To alleviate these problems, we introduce a novel representation, extended bounding field-of-view (eBFoV), for target localization and use it as the foundation of a general 360 tracking framework which is applicable for both omnidirectional visual object tracking and segmentation tasks. Building upon our previous work on omnidirectional visual object tracking (360VOT), we propose a comprehensive dataset and benchmark that incorporates a new component called omnidirectional video object segmentation (360VOS). The 360VOS dataset includes 290 sequences accompanied by dense pixel-wise masks and covers a broader range of target categories. To support both the development and evaluation of algorithms in this domain, we divide the dataset into a training subset with 170 sequences and a testing subset with 120 sequences. Furthermore, we tailor evaluation metrics for both omnidirectional tracking and segmentation to ensure rigorous assessment. Through extensive experiments, we benchmark state-of-the-art approaches and demonstrate the effectiveness of our proposed 360 tracking framework and training dataset. Homepage: https://360vots.hkustvgd.com/