Video Reenactment as Inductive Bias for Content-Motion Disentanglement

TL;DR

This paper introduces MTC-VAE, a self-supervised model for disentangling motion and content in videos using chunk-wise analysis and a novel reenactment loss, leading to improved video reconstruction and motion transfer.

Contribution

The paper proposes a new chunk-wise VAE approach with a Blind Reenactment Loss for better motion-content disentanglement in videos, outperforming existing methods.

Findings

Outperforms existing methods in disentanglement metrics

Achieves higher quality video reconstructions

Demonstrates effective motion transfer in reenactment tasks

Abstract

Independent components within low-dimensional representations are essential inputs in several downstream tasks, and provide explanations over the observed data. Video-based disentangled factors of variation provide low-dimensional representations that can be identified and used to feed task-specific models. We introduce MTC-VAE, a self-supervised motion-transfer VAE model to disentangle motion and content from videos. Unlike previous work on video content-motion disentanglement, we adopt a chunk-wise modeling approach and take advantage of the motion information contained in spatiotemporal neighborhoods. Our model yields independent per-chunk representations that preserve temporal consistency. Hence, we reconstruct whole videos in a single forward-pass. We extend the ELBO's log-likelihood term and include a Blind Reenactment Loss as an inductive bias to leverage motion disentanglement, under the assumption that swapping motion features yields reenactment between two videos. We evaluate our model with recently-proposed disentanglement metrics and show that it outperforms a variety of methods for video motion-content disentanglement. Experiments on video reenactment show the effectiveness of our disentanglement in the input space where our model outperforms the baselines in reconstruction quality and motion alignment.