Unsupervised Learning of Long-Term Motion Dynamics for Videos

TL;DR

This paper introduces an unsupervised method for learning long-term motion dynamics in videos by predicting sequences of atomic 3D flows, enabling robust activity recognition across multiple modalities and datasets.

Contribution

It proposes a novel unsupervised framework using RGB-D data and RNNs to encode long-term motion dependencies, improving video representation for activity classification.

Findings

Effective in activity classification on NTU RGB+D and MSR datasets.

Works across RGB, Depth, and RGB-D modalities.

Learns robust temporal representations without supervision.

Abstract

We present an unsupervised representation learning approach that compactly encodes the motion dependencies in videos. Given a pair of images from a video clip, our framework learns to predict the long-term 3D motions. To reduce the complexity of the learning framework, we propose to describe the motion as a sequence of atomic 3D flows computed with RGB-D modality. We use a Recurrent Neural Network based Encoder-Decoder framework to predict these sequences of flows. We argue that in order for the decoder to reconstruct these sequences, the encoder must learn a robust video representation that captures long-term motion dependencies and spatial-temporal relations. We demonstrate the effectiveness of our learned temporal representations on activity classification across multiple modalities and datasets such as NTU RGB+D and MSR Daily Activity 3D. Our framework is generic to any input modality, i.e., RGB, Depth, and RGB-D videos.