Unsupervised Video Understanding by Reconciliation of Posture Similarities

TL;DR

This paper introduces an unsupervised deep learning method for understanding human activities in videos by learning posture representations without manual annotations, enabling retrieval, super-resolution, and frame synthesis.

Contribution

It presents a novel unsupervised approach that combines sequence matching and CNNs to learn structured posture embeddings from raw video data.

Findings

Learns posture representations without supervision

Enables posture retrieval and temporal super-resolution

Allows frame synthesis based on learned embeddings

Abstract

Understanding human activity and being able to explain it in detail surpasses mere action classification by far in both complexity and value. The challenge is thus to describe an activity on the basis of its most fundamental constituents, the individual postures and their distinctive transitions. Supervised learning of such a fine-grained representation based on elementary poses is very tedious and does not scale. Therefore, we propose a completely unsupervised deep learning procedure based solely on video sequences, which starts from scratch without requiring pre-trained networks, predefined body models, or keypoints. A combinatorial sequence matching algorithm proposes relations between frames from subsets of the training data, while a CNN is reconciling the transitivity conflicts of the different subsets to learn a single concerted pose embedding despite changes in appearance across sequences. Without any manual annotation, the model learns a structured representation of postures and their temporal development. The model not only enables retrieval of similar postures but also temporal super-resolution. Additionally, based on a recurrent formulation, next frames can be synthesized.