360-Degree Full-view Image Segmentation by Spherical Convolution compatible with Large-scale Planar Pre-trained Models

TL;DR

This paper presents a spherical sampling method that allows existing 2D pre-trained models to effectively perform panoramic image segmentation by addressing distortions and discontinuities in panoramic images.

Contribution

The paper introduces a novel spherical sampling technique compatible with large-scale planar pre-trained models for panoramic image segmentation.

Findings

Effective mitigation of distortions in panoramic images

Improved segmentation performance on Stanford2D3D dataset

Compatibility with existing 2D pre-trained models

Abstract

Due to the current lack of large-scale datasets at the million-scale level, tasks involving panoramic images predominantly rely on existing two-dimensional pre-trained image benchmark models as backbone networks. However, these networks are not equipped to recognize the distortions and discontinuities inherent in panoramic images, which adversely affects their performance in such tasks. In this paper, we introduce a novel spherical sampling method for panoramic images that enables the direct utilization of existing pre-trained models developed for two-dimensional images. Our method employs spherical discrete sampling based on the weights of the pre-trained models, effectively mitigating distortions while achieving favorable initial training values. Additionally, we apply the proposed sampling method to panoramic image segmentation, utilizing features obtained from the spherical model as masks for specific channel attentions, which yields commendable results on commonly used indoor datasets, Stanford2D3D.