Semantic Motion Segmentation Using Dense CRF Formulation

TL;DR

This paper introduces a joint semantic and motion segmentation method using a dense CRF model that leverages semantic, geometric, and optical flow cues, improving accuracy on outdoor datasets.

Contribution

It proposes a novel joint inference algorithm for semantic and motion labels using a dense CRF with integrated constraints, outperforming existing methods.

Findings

Improved semantic and motion segmentation accuracy on KITTI dataset.

Outperforms recent motion detection algorithms.

Enhances semantic labeling for critical object classes like pedestrians and cars.

Abstract

While the literature has been fairly dense in the areas of scene understanding and semantic labeling there have been few works that make use of motion cues to embellish semantic performance and vice versa. In this paper, we address the problem of semantic motion segmentation, and show how semantic and motion priors augments performance. We pro- pose an algorithm that jointly infers the semantic class and motion labels of an object. Integrating semantic, geometric and optical ow based constraints into a dense CRF-model we infer both the object class as well as motion class, for each pixel. We found improvement in performance using a fully connected CRF as compared to a standard clique-based CRFs. For inference, we use a Mean Field approximation based algorithm. Our method outperforms recently pro- posed motion detection algorithms and also improves the semantic labeling compared to the state-of-the-art Automatic Labeling Environment algorithm on the challenging KITTI dataset especially for object classes such as pedestrians and cars that are critical to an outdoor robotic navigation scenario.