Occlusion-Aware Seamless Segmentation

TL;DR

This paper introduces Occlusion-Aware Seamless Segmentation (OASS), a new task addressing panoramic image challenges through a novel dataset and a solution called UnmaskFormer, which unifies occlusion handling, domain adaptation, and segmentation.

Contribution

The paper presents the first dataset BlendPASS and a novel method UnmaskFormer for occlusion-aware panoramic segmentation, advancing the state-of-the-art in this area.

Findings

UnmaskFormer achieves 26.58% mAPQ and 43.66% mIoU on BlendPASS.

Outperforms previous methods on SynPASS and DensePASS datasets.

Provides a new benchmark and source code for future research.

Abstract

Panoramic images can broaden the Field of View (FoV), occlusion-aware prediction can deepen the understanding of the scene, and domain adaptation can transfer across viewing domains. In this work, we introduce a novel task, Occlusion-Aware Seamless Segmentation (OASS), which simultaneously tackles all these three challenges. For benchmarking OASS, we establish a new human-annotated dataset for Blending Panoramic Amodal Seamless Segmentation, i.e., BlendPASS. Besides, we propose the first solution UnmaskFormer, aiming at unmasking the narrow FoV, occlusions, and domain gaps all at once. Specifically, UnmaskFormer includes the crucial designs of Unmasking Attention (UA) and Amodal-oriented Mix (AoMix). Our method achieves state-of-the-art performance on the BlendPASS dataset, reaching a remarkable mAPQ of 26.58% and mIoU of 43.66%. On public panoramic semantic segmentation datasets, i.e., SynPASS and DensePASS, our method outperforms previous methods and obtains 45.34% and 48.08% in mIoU, respectively. The fresh BlendPASS dataset and our source code are available at https://github.com/yihong-97/OASS.