Exploiting video sequences for unsupervised disentangling in generative adversarial networks

TL;DR

This paper introduces an unsupervised method for training GANs to learn a disentangled latent space from video sequences, separating content and motion attributes without supervision.

Contribution

It proposes a simple modification to standard GAN training that leverages video correlations to achieve disentanglement of content and motion in the latent space.

Findings

Successfully disentangles identity and motion attributes in face videos

Works on datasets with different subjects and movements

Provides both qualitative and quantitative evidence of disentanglement

Abstract

In this work we present an adversarial training algorithm that exploits correlations in video to learn --without supervision-- an image generator model with a disentangled latent space. The proposed methodology requires only a few modifications to the standard algorithm of Generative Adversarial Networks (GAN) and involves training with sets of frames taken from short videos. We train our model over two datasets of face-centered videos which present different people speaking or moving the head: VidTIMIT and YouTube Faces datasets. We found that our proposal allows us to split the generator latent space into two subspaces. One of them controls content attributes, those that do not change along short video sequences. For the considered datasets, this is the identity of the generated face. The other subspace controls motion attributes, those attributes that are observed to change along short videos. We observed that these motion attributes are face expressions, head orientation, lips and eyes movement. The presented experiments provide quantitative and qualitative evidence supporting that the proposed methodology induces a disentangling of this two kinds of attributes in the latent space.