Training of Convolutional Networks on Multiple Heterogeneous Datasets for Street Scene Semantic Segmentation

TL;DR

This paper introduces a hierarchical convolutional network trained on multiple heterogeneous datasets for improved street scene semantic segmentation, handling various annotation types and semantic levels.

Contribution

It is the first to train a single network on three diverse datasets with different annotation types and semantic hierarchies for street scene segmentation.

Findings

Achieved 13% improvement in mean pixel accuracy on Cityscapes

Improved accuracy by 2.4% on Vistas dataset

Inferred at 17 fps on GPU for 108 classes

Abstract

We propose a convolutional network with hierarchical classifiers for per-pixel semantic segmentation, which is able to be trained on multiple, heterogeneous datasets and exploit their semantic hierarchy. Our network is the first to be simultaneously trained on three different datasets from the intelligent vehicles domain, i.e. Cityscapes, GTSDB and Mapillary Vistas, and is able to handle different semantic level-of-detail, class imbalances, and different annotation types, i.e. dense per-pixel and sparse bounding-box labels. We assess our hierarchical approach, by comparing against flat, non-hierarchical classifiers and we show improvements in mean pixel accuracy of 13.0% for Cityscapes classes and 2.4% for Vistas classes and 32.3% for GTSDB classes. Our implementation achieves inference rates of 17 fps at a resolution of 520x706 for 108 classes running on a GPU.