Self-supervised learning of class embeddings from video

TL;DR

This paper presents a self-supervised learning approach for video-based class embeddings that encode pose and shape, enabling effective downstream tasks and achieving state-of-the-art results without supervision.

Contribution

Introduces a hierarchical probabilistic decoder for learning class-specific embeddings from videos, generalizing across deformable object classes and outperforming existing self-supervised methods.

Findings

Achieves state-of-the-art performance on multiple deformable object classes.

Embeddings generalize well across different domains.

Approaches supervised performance levels without using labels.

Abstract

This work explores how to use self-supervised learning on videos to learn a class-specific image embedding that encodes pose and shape information. At train time, two frames of the same video of an object class (e.g. human upper body) are extracted and each encoded to an embedding. Conditioned on these embeddings, the decoder network is tasked to transform one frame into another. To successfully perform long range transformations (e.g. a wrist lowered in one image should be mapped to the same wrist raised in another), we introduce a hierarchical probabilistic network decoder model. Once trained, the embedding can be used for a variety of downstream tasks and domains. We demonstrate our approach quantitatively on three distinct deformable object classes -- human full bodies, upper bodies, faces -- and show experimentally that the learned embeddings do indeed generalise. They achieve state-of-the-art performance in comparison to other self-supervised methods trained on the same datasets, and approach the performance of fully supervised methods.