Unsupervised Learning of Disentangled Representations from Video

TL;DR

This paper introduces DrNET, an unsupervised model that learns disentangled video representations by separating stationary and dynamic parts, enabling future frame prediction and coherent video generation.

Contribution

The paper proposes a novel adversarial approach to learn disentangled representations from video, leveraging temporal coherence for improved future frame prediction.

Findings

Successfully disentangles stationary and dynamic components in videos

Enables accurate long-term future frame prediction

Demonstrates effectiveness on synthetic and real videos

Abstract

We present a new model DrNET that learns disentangled image representations from video. Our approach leverages the temporal coherence of video and a novel adversarial loss to learn a representation that factorizes each frame into a stationary part and a temporally varying component. The disentangled representation can be used for a range of tasks. For example, applying a standard LSTM to the time-vary components enables prediction of future frames. We evaluate our approach on a range of synthetic and real videos, demonstrating the ability to coherently generate hundreds of steps into the future.