Multi Modal Semantic Segmentation using Synthetic Data

TL;DR

This paper explores semantic segmentation of 3D scenes using synthetic data to learn geometric and texture cues, and evaluates the transferability of this knowledge to real-world datasets.

Contribution

It introduces a method to train a neural network on synthetic 3D scenes for semantic classification without real-world labels, demonstrating effective transfer to real data.

Findings

Neural network learns geometric context from synthetic scenes

Model successfully applies learned features to real-world data

Synthetic training improves 3D scene understanding

Abstract

Semantic understanding of scenes in three-dimensional space (3D) is a quintessential part of robotics oriented applications such as autonomous driving as it provides geometric cues such as size, orientation and true distance of separation to objects which are crucial for taking mission critical decisions. As a first step, in this work we investigate the possibility of semantically classifying different parts of a given scene in 3D by learning the underlying geometric context in addition to the texture cues BUT in the absence of labelled real-world datasets. To this end we generate a large number of synthetic scenes, their pixel-wise labels and corresponding 3D representations using CARLA software framework. We then build a deep neural network that learns underlying category specific 3D representation and texture cues from color information of the rendered synthetic scenes. Further on we apply the learned model on different real world datasets to evaluate its performance. Our preliminary investigation of results show that the neural network is able to learn the geometric context from synthetic scenes and effectively apply this knowledge to classify each point of a 3D representation of a scene in real-world.