Single Frame Semantic Segmentation Using Multi-Modal Spherical Images

TL;DR

This paper introduces a transformer-based multi-modal fusion method for single-frame semantic segmentation of panoramic images, effectively handling distortions and improving scene understanding across various indoor datasets.

Contribution

It presents a novel cross-modal fusion architecture with distortion-aware modules for omnidirectional scene perception, outperforming existing methods on multiple datasets.

Findings

Achieved state-of-the-art mIoU scores on three indoor panoramic datasets.

Effectively addressed panorama distortions with distortion-aware modules.

Enhanced multi-modal feature communication for improved segmentation accuracy.

Abstract

In recent years, the research community has shown a lot of interest to panoramic images that offer a 360-degree directional perspective. Multiple data modalities can be fed, and complimentary characteristics can be utilized for more robust and rich scene interpretation based on semantic segmentation, to fully realize the potential. Existing research, however, mostly concentrated on pinhole RGB-X semantic segmentation. In this study, we propose a transformer-based cross-modal fusion architecture to bridge the gap between multi-modal fusion and omnidirectional scene perception. We employ distortion-aware modules to address extreme object deformations and panorama distortions that result from equirectangular representation. Additionally, we conduct cross-modal interactions for feature rectification and information exchange before merging the features in order to communicate long-range contexts for bi-modal and tri-modal feature streams. In thorough tests using combinations of four different modality types in three indoor panoramic-view datasets, our technique achieved state-of-the-art mIoU performance: 60.60% on Stanford2D3DS (RGB-HHA), 71.97% Structured3D (RGB-D-N), and 35.92% Matterport3D (RGB-D). We plan to release all codes and trained models soon.