Learning segmentation from point trajectories

TL;DR

This paper introduces a novel method for video object segmentation that leverages long-term point trajectories and a low-rank subspace clustering approach to improve motion-based segmentation accuracy without supervision.

Contribution

It proposes a new loss function inspired by subspace clustering to utilize long-term trajectories for better segmentation, addressing the complexity of modeling long-term motion.

Findings

Outperforms prior methods on motion segmentation benchmarks.

Demonstrates the utility of long-term motion information.

Validates the effectiveness of low-rank trajectory grouping.

Abstract

We consider the problem of segmenting objects in videos based on their motion and no other forms of supervision. Prior work has often approached this problem by using the principle of common fate, namely the fact that the motion of points that belong to the same object is strongly correlated. However, most authors have only considered instantaneous motion from optical flow. In this work, we present a way to train a segmentation network using long-term point trajectories as a supervisory signal to complement optical flow. The key difficulty is that long-term motion, unlike instantaneous motion, is difficult to model -- any parametric approximation is unlikely to capture complex motion patterns over long periods of time. We instead draw inspiration from subspace clustering approaches, proposing a loss function that seeks to group the trajectories into low-rank matrices where the motion of object points can be approximately explained as a linear combination of other point tracks. Our method outperforms the prior art on motion-based segmentation, which shows the utility of long-term motion and the effectiveness of our formulation.