3D Geometry-Aware Semantic Labeling of Outdoor Street Scenes

TL;DR

This paper introduces a 3D geometry-aware semantic labeling method for outdoor street scenes that leverages 3D convolution in a residual voxel network, outperforming existing approaches on standard datasets.

Contribution

It proposes a novel 3D convolutional approach for semantic labeling that effectively incorporates 3D geometric information, improving accuracy over prior methods.

Findings

Outperforms state-of-the-art methods on Synthia and Cityscape datasets.

Demonstrates the effectiveness of 3D convolution in semantic labeling.

Shows simple 3D representation enhances geometric information utilization.

Abstract

This paper is concerned with the problem of how to better exploit 3D geometric information for dense semantic image labeling. Existing methods often treat the available 3D geometry information (e.g., 3D depth-map) simply as an additional image channel besides the R-G-B color channels, and apply the same technique for RGB image labeling. In this paper, we demonstrate that directly performing 3D convolution in the framework of a residual connected 3D voxel top-down modulation network can lead to superior results. Specifically, we propose a 3D semantic labeling method to label outdoor street scenes whenever a dense depth map is available. Experiments on the "Synthia" and "Cityscape" datasets show our method outperforms the state-of-the-art methods, suggesting such a simple 3D representation is effective in incorporating 3D geometric information.