Capturing Omni-Range Context for Omnidirectional Segmentation

TL;DR

This paper introduces ECANets, a novel attention-based model designed for omnidirectional image segmentation, significantly improving performance on 360-degree street scene data through multi-source training and a new dataset.

Contribution

The paper presents ECANets, a new model with attention mechanisms for omnidirectional segmentation, and introduces WildPASS, a diverse panoramic dataset, achieving state-of-the-art results.

Findings

Achieved 60.2% mIoU on PASS benchmark.

Achieved 69.0% mIoU on WildPASS dataset.

Enhanced segmentation performance with multi-source omni-supervised training.

Abstract

Convolutional Networks (ConvNets) excel at semantic segmentation and have become a vital component for perception in autonomous driving. Enabling an all-encompassing view of street-scenes, omnidirectional cameras present themselves as a perfect fit in such systems. Most segmentation models for parsing urban environments operate on common, narrow Field of View (FoV) images. Transferring these models from the domain they were designed for to 360-degree perception, their performance drops dramatically, e.g., by an absolute 30.0% (mIoU) on established test-beds. To bridge the gap in terms of FoV and structural distribution between the imaging domains, we introduce Efficient Concurrent Attention Networks (ECANets), directly capturing the inherent long-range dependencies in omnidirectional imagery. In addition to the learned attention-based contextual priors that can stretch across 360-degree images, we upgrade model training by leveraging multi-source and omni-supervised learning, taking advantage of both: Densely labeled and unlabeled data originating from multiple datasets. To foster progress in panoramic image segmentation, we put forward and extensively evaluate models on Wild PAnoramic Semantic Segmentation (WildPASS), a dataset designed to capture diverse scenes from all around the globe. Our novel model, training regimen and multi-source prediction fusion elevate the performance (mIoU) to new state-of-the-art results on the public PASS (60.2%) and the fresh WildPASS (69.0%) benchmarks.