Three for one and one for three: Flow, Segmentation, and Surface Normals

TL;DR

This paper investigates how optical flow, semantic segmentation, and surface normals interact and improve scene understanding when combined, using a modular network and a synthetic dataset.

Contribution

It introduces a modular convolutional network trained on a synthetic dataset to study the influence and synergy of three scene understanding modalities.

Findings

Positive influence among modalities on object boundaries

Enhanced region consistency and scene structure understanding

Modular approach improves joint feature learning

Abstract

Optical flow, semantic segmentation, and surface normals represent different information modalities, yet together they bring better cues for scene understanding problems. In this paper, we study the influence between the three modalities: how one impacts on the others and their efficiency in combination. We employ a modular approach using a convolutional refinement network which is trained supervised but isolated from RGB images to enforce joint modality features. To assist the training process, we create a large-scale synthetic outdoor dataset that supports dense annotation of semantic segmentation, optical flow, and surface normals. The experimental results show positive influence among the three modalities, especially for objects' boundaries, region consistency, and scene structures.