Technical Report: Co-learning of geometry and semantics for online 3D mapping

TL;DR

This paper presents a multi-task neural network for joint depth and semantic segmentation to improve 3D semantic reconstruction for autonomous navigation, demonstrating superior performance on the 3DRMS Challenge dataset.

Contribution

It introduces a deep multi-task neural network that refines depth and semantic segmentation simultaneously for enhanced 3D mapping.

Findings

Outperforms state-of-the-art methods on the 3DRMS dataset

Produces accurate semantic 3D point clouds and meshes

Effective co-learning improves reconstruction quality

Abstract

This paper is a technical report about our submission for the ECCV 2018 3DRMS Workshop Challenge on Semantic 3D Reconstruction \cite{Tylecek2018rms}. In this paper, we address 3D semantic reconstruction for autonomous navigation using co-learning of depth map and semantic segmentation. The core of our pipeline is a deep multi-task neural network which tightly refines depth and also produces accurate semantic segmentation maps. Its inputs are an image and a raw depth map produced from a pair of images by standard stereo vision. The resulting semantic 3D point clouds are then merged in order to create a consistent 3D mesh, in turn used to produce dense semantic 3D reconstruction maps. The performances of each step of the proposed method are evaluated on the dataset and multiple tasks of the 3DRMS Challenge, and repeatedly surpass state-of-the-art approaches.