Open Panoramic Segmentation

TL;DR

This paper introduces Open Panoramic Segmentation (OPS), a zero-shot task enabling models trained on restricted FoV images to segment panoramic images, with a novel model OOOPS and a distortion-aware data augmentation RERP, achieving state-of-the-art results.

Contribution

The paper defines the new OPS task, proposes the OOOPS model with Deformable Adapter Network, and introduces RERP data augmentation to improve zero-shot panoramic segmentation.

Findings

Achieved +2.2% mIoU on WildPASS dataset.

Achieved +2.4% mIoU on Stanford2D3D dataset.

Outperformed existing open-vocabulary segmentation methods.

Abstract

Panoramic images, capturing a 360{\deg} field of view (FoV), encompass omnidirectional spatial information crucial for scene understanding. However, it is not only costly to obtain training-sufficient dense-annotated panoramas but also application-restricted when training models in a close-vocabulary setting. To tackle this problem, in this work, we define a new task termed Open Panoramic Segmentation (OPS), where models are trained with FoV-restricted pinhole images in the source domain in an open-vocabulary setting while evaluated with FoV-open panoramic images in the target domain, enabling the zero-shot open panoramic semantic segmentation ability of models. Moreover, we propose a model named OOOPS with a Deformable Adapter Network (DAN), which significantly improves zero-shot panoramic semantic segmentation performance. To further enhance the distortion-aware modeling ability from the pinhole source domain, we propose a novel data augmentation method called Random Equirectangular Projection (RERP) which is specifically designed to address object deformations in advance. Surpassing other state-of-the-art open-vocabulary semantic segmentation approaches, a remarkable performance boost on three panoramic datasets, WildPASS, Stanford2D3D, and Matterport3D, proves the effectiveness of our proposed OOOPS model with RERP on the OPS task, especially +2.2% on outdoor WildPASS and +2.4% mIoU on indoor Stanford2D3D. The source code is publicly available at https://junweizheng93.github.io/publications/OPS/OPS.html.