REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion

TL;DR

REL-SF4PASS introduces a novel depth representation and fusion method for panoramic semantic segmentation, significantly enhancing accuracy and robustness by better utilizing spherical geometry and multi-modal data.

Contribution

The paper proposes REL depth representation and Spherical-dynamic Multi-Modal Fusion, improving panoramic segmentation by fully exploiting spherical geometry and reducing projection issues.

Findings

Achieves 2.35% higher mIoU on Stanford2D3D datasets.

Reduces performance variance by approximately 70% under 3D disturbances.

Enhances robustness and accuracy in panoramic scene understanding.

Abstract

As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to give complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or using the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and Spherical-dynamic Multi-Modal Fusion SMMF. REL is made up of Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle, which fully represents 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, which uses different fusion strategies to match the different regions in panoramic images. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmark, Stanford2D3D Panoramic datasets. It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance.