Boundary Flow: A Siamese Network that Predicts Boundary Motion without Training on Motion

TL;DR

This paper introduces Boundary Flow, a Siamese network that predicts boundary motion in videos without requiring motion training, improving boundary detection and optical flow estimation.

Contribution

It proposes a novel fully convolutional Siamese network for joint boundary detection and motion estimation, with an unconventional deconvolution approach and edgelet-based filtering.

Findings

Achieves state-of-the-art boundary detection on VSB100.

Presents first boundary flow results on Sintel dataset.

Improves optical flow estimation by incorporating boundary matches.

Abstract

Using deep learning, this paper addresses the problem of joint object boundary detection and boundary motion estimation in videos, which we named boundary flow estimation. Boundary flow is an important mid-level visual cue as boundaries characterize objects spatial extents, and the flow indicates objects motions and interactions. Yet, most prior work on motion estimation has focused on dense object motion or feature points that may not necessarily reside on boundaries. For boundary flow estimation, we specify a new fully convolutional Siamese network (FCSN) that jointly estimates object-level boundaries in two consecutive frames. Boundary correspondences in the two frames are predicted by the same FCSN with a new, unconventional deconvolution approach. Finally, the boundary flow estimate is improved with an edgelet-based filtering. Evaluation is conducted on three tasks: boundary detection in videos, boundary flow estimation, and optical flow estimation. On boundary detection, we achieve the state-of-the-art performance on the benchmark VSB100 dataset. On boundary flow estimation, we present the first results on the Sintel training dataset. For optical flow estimation, we run the recent approach CPMFlow but on the augmented input with our boundary-flow matches, and achieve significant performance improvement on the Sintel benchmark.