Self-Supervision by Prediction for Object Discovery in Videos

TL;DR

This paper introduces a self-supervised, object-centric model for video prediction that disentangles objects and motion, handles occlusion, and does not require manual annotations, advancing unsupervised learning in videos.

Contribution

It presents a novel self-supervised framework for object discovery and prediction in videos, explicitly modeling occlusion and background in an unsupervised manner.

Findings

Effective disentanglement of objects and motion dynamics

Handles occlusion and inpaints inferred objects

Promising results in object-centric video prediction

Abstract

Despite their irresistible success, deep learning algorithms still heavily rely on annotated data. On the other hand, unsupervised settings pose many challenges, especially about determining the right inductive bias in diverse scenarios. One scalable solution is to make the model generate the supervision for itself by leveraging some part of the input data, which is known as self-supervised learning. In this paper, we use the prediction task as self-supervision and build a novel object-centric model for image sequence representation. In addition to disentangling the notion of objects and the motion dynamics, our compositional structure explicitly handles occlusion and inpaints inferred objects and background for the composition of the predicted frame. With the aid of auxiliary loss functions that promote spatially and temporally consistent object representations, our self-supervised framework can be trained without the help of any manual annotation or pretrained network. Initial experiments confirm that the proposed pipeline is a promising step towards object-centric video prediction.