Scale Invariant Semantic Segmentation with RGB-D Fusion

TL;DR

This paper introduces a scale-invariant semantic segmentation neural network that fuses RGB and depth data, improving segmentation accuracy across varying object scales in outdoor scenes, validated on Cityscapes and synthetic datasets.

Contribution

It presents a novel fusion block for combining RGB and depth features in a scale-invariant segmentation model based on DeepLab-v2, enhancing performance on outdoor scenes.

Findings

Comparable to state-of-the-art on Cityscapes

Effective on real and synthetic datasets

Improves segmentation of multi-scale objects

Abstract

In this paper, we propose a neural network architecture for scale-invariant semantic segmentation using RGB-D images. We utilize depth information as an additional modality apart from color images only. Especially in an outdoor scene which consists of different scale objects due to the distance of the objects from the camera. The near distance objects consist of significantly more pixels than the far ones. We propose to incorporate depth information to the RGB data for pixel-wise semantic segmentation to address the different scale objects in an outdoor scene. We adapt to a well-known DeepLab-v2(ResNet-101) model as our RGB baseline. Depth images are passed separately as an additional input with a distinct branch. The intermediate feature maps of both color and depth image branch are fused using a novel fusion block. Our model is compact and can be easily applied to the other RGB model. We perform extensive qualitative and quantitative evaluation on a challenging dataset Cityscapes. The results obtained are comparable to the state-of-the-art. Additionally, we evaluated our model on a self-recorded real dataset. For the shake of extended evaluation of a driving scene with ground truth we generated a synthetic dataset using popular vehicle simulation project CARLA. The results obtained from the real and synthetic dataset shows the effectiveness of our approach.