SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

TL;DR

SGAT4PASS introduces a spherical geometry-aware transformer that enhances panoramic semantic segmentation by effectively handling 3D disturbances, leading to improved accuracy and robustness on panoramic datasets.

Contribution

It proposes a novel spherical geometry-aware framework with three modules to better handle 3D disturbances in panoramic images for semantic segmentation.

Findings

Achieves approximately 2% increase in mIoU on Stanford2D3D dataset.

Significantly improves robustness against small 3D disturbances.

Enhances performance stability in panoramic semantic segmentation.

Abstract

As an important and challenging problem in computer vision, PAnoramic Semantic Segmentation (PASS) gives complete scene perception based on an ultra-wide angle of view. Usually, prevalent PASS methods with 2D panoramic image input focus on solving image distortions but lack consideration of the 3D properties of original $360^{\circ}$ data. Therefore, their performance will drop a lot when inputting panoramic images with the 3D disturbance. To be more robust to 3D disturbance, we propose our Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation (SGAT4PASS), considering 3D spherical geometry knowledge. Specifically, a spherical geometry-aware framework is proposed for PASS. It includes three modules, i.e., spherical geometry-aware image projection, spherical deformable patch embedding, and a panorama-aware loss, which takes input images with 3D disturbance into account, adds a spherical geometry-aware constraint on the existing deformable patch embedding, and indicates the pixel density of original $360^{\circ}$ data, respectively. Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS significantly improves performance and robustness, with approximately a 2% increase in mIoU, and when small 3D disturbances occur in the data, the stability of our performance is improved by an order of magnitude. Our code and supplementary material are available at https://github.com/TencentARC/SGAT4PASS.