Self-Supervised Learning via Conditional Motion Propagation

TL;DR

This paper introduces a novel self-supervised learning method using conditional motion propagation, which effectively learns visual representations from motion cues by resolving ambiguity with sparse guidance, outperforming previous methods on various tasks.

Contribution

The work proposes a new pretext task of conditional motion propagation that improves motion understanding and feature learning without strong assumptions on object motion.

Findings

Achieves state-of-the-art results on semantic segmentation, instance segmentation, and human parsing.

Learns structurally coherent and expressive features from motion cues.

Extends to applications like semi-automatic pixel-level annotation.

Abstract

Intelligent agent naturally learns from motion. Various self-supervised algorithms have leveraged motion cues to learn effective visual representations. The hurdle here is that motion is both ambiguous and complex, rendering previous works either suffer from degraded learning efficacy, or resort to strong assumptions on object motions. In this work, we design a new learning-from-motion paradigm to bridge these gaps. Instead of explicitly modeling the motion probabilities, we design the pretext task as a conditional motion propagation problem. Given an input image and several sparse flow guidance vectors on it, our framework seeks to recover the full-image motion. Compared to other alternatives, our framework has several appealing properties: (1) Using sparse flow guidance during training resolves the inherent motion ambiguity, and thus easing feature learning. (2) Solving the pretext task of conditional motion propagation encourages the emergence of kinematically-sound representations that poss greater expressive power. Extensive experiments demonstrate that our framework learns structural and coherent features; and achieves state-of-the-art self-supervision performance on several downstream tasks including semantic segmentation, instance segmentation, and human parsing. Furthermore, our framework is successfully extended to several useful applications such as semi-automatic pixel-level annotation. Project page: "http://mmlab.ie.cuhk.edu.hk/projects/CMP/".