USegScene: Unsupervised Learning of Depth, Optical Flow and Ego-Motion with Semantic Guidance and Coupled Networks

TL;DR

USegScene introduces a semantically guided, coupled neural network framework for unsupervised learning of depth, optical flow, and ego-motion, significantly improving accuracy on KITTI dataset by leveraging semantic information and joint predictions.

Contribution

The paper presents a novel architecture that jointly predicts depth, optical flow, and ego-motion with semantic guidance, enabling cross-task information sharing and explicit occlusion learning.

Findings

Outperforms existing methods on KITTI dataset

Joint prediction improves accuracy of depth and flow estimation

Semantic guidance enhances regularization and occlusion handling

Abstract

In this paper we propose USegScene, a framework for semantically guided unsupervised learning of depth, optical flow and ego-motion estimation for stereo camera images using convolutional neural networks. Our framework leverages semantic information for improved regularization of depth and optical flow maps, multimodal fusion and occlusion filling considering dynamic rigid object motions as independent SE(3) transformations. Furthermore, complementary to pure photo-metric matching, we propose matching of semantic features, pixel-wise classes and object instance borders between the consecutive images. In contrast to previous methods, we propose a network architecture that jointly predicts all outputs using shared encoders and allows passing information across the task-domains, e.g., the prediction of optical flow can benefit from the prediction of the depth. Furthermore, we explicitly learn the depth and optical flow occlusion maps inside the network, which are leveraged in order to improve the predictions in therespective regions. We present results on the popular KITTI dataset and show that our approach outperforms other methods by a large margin.