Joint Detection of Motion Boundaries and Occlusions

TL;DR

MONet is a convolutional neural network that jointly detects motion boundaries and occlusion regions in video, effectively handling discontinuities and undefined flows by using a cost block approach and bidirectional reasoning.

Contribution

The paper introduces MONet, a novel CNN architecture that jointly detects motion boundaries and occlusions using a computationally efficient cost block and bidirectional reasoning, outperforming prior methods.

Findings

MONet outperforms previous state-of-the-art on Sintel and FlyingChairsOcc benchmarks.

The use of a two-dimensional cost block reduces computational expense compared to four-dimensional volumes.

Arranging decoder layers fine-to-coarse improves detection performance.

Abstract

We propose MONet, a convolutional neural network that jointly detects motion boundaries (MBs) and occlusion regions (Occs) in video both forward and backward in time. Detection is difficult because optical flow is discontinuous along MBs and undefined in Occs, while many flow estimators assume smoothness and a flow defined everywhere. To reason in the two time directions simultaneously, we direct-warp the estimated maps between the two frames. Since appearance mismatches between frames often signal vicinity to MBs or Occs, we construct a cost block that for each feature in one frame records the lowest discrepancy with matching features in a search range. This cost block is two-dimensional, and much less expensive than the four-dimensional cost volumes used in flow analysis. Cost-block features are computed by an encoder, and MB and Occ estimates are computed by a decoder. We found that arranging decoder layers fine-to-coarse, rather than coarse-to-fine, improves performance. MONet outperforms the prior state of the art for both tasks on the Sintel and FlyingChairsOcc benchmarks without any fine-tuning on them.