Monocular Depth Estimation by Learning from Heterogeneous Datasets

TL;DR

This paper introduces a method for monocular depth estimation that learns from heterogeneous datasets containing different types of annotations, improving performance over existing methods.

Contribution

It demonstrates that CNNs for depth estimation can be trained using separate datasets for depth and semantic labels, removing the need for joint annotations.

Findings

Outperforms state-of-the-art on KITTI and Cityscapes datasets.

Shows that training with heterogeneous datasets is effective.

Reduces annotation constraints for monocular depth estimation.

Abstract

Depth estimation provides essential information to perform autonomous driving and driver assistance. Especially, Monocular Depth Estimation is interesting from a practical point of view, since using a single camera is cheaper than many other options and avoids the need for continuous calibration strategies as required by stereo-vision approaches. State-of-the-art methods for Monocular Depth Estimation are based on Convolutional Neural Networks (CNNs). A promising line of work consists of introducing additional semantic information about the traffic scene when training CNNs for depth estimation. In practice, this means that the depth data used for CNN training is complemented with images having pixel-wise semantic labels, which usually are difficult to annotate (e.g. crowded urban images). Moreover, so far it is common practice to assume that the same raw training data is associated with both types of ground truth, i.e., depth and semantic labels. The main contribution of this paper is to show that this hard constraint can be circumvented, i.e., that we can train CNNs for depth estimation by leveraging the depth and semantic information coming from heterogeneous datasets. In order to illustrate the benefits of our approach, we combine KITTI depth and Cityscapes semantic segmentation datasets, outperforming state-of-the-art results on Monocular Depth Estimation.