Mutual Information Based Method for Unsupervised Disentanglement of Video Representation

TL;DR

This paper introduces MIPAE, a framework that disentangles video representations into content and pose variables using mutual information, improving future frame prediction in videos.

Contribution

The work proposes a novel mutual information loss and a disentanglement metric, advancing unsupervised video representation learning.

Findings

MIPAE effectively disentangles content and pose in video data.

Disentangled representations improve future frame prediction quality.

MIG scores align with visual and quantitative evaluation metrics.

Abstract

Video Prediction is an interesting and challenging task of predicting future frames from a given set context frames that belong to a video sequence. Video prediction models have found prospective applications in Maneuver Planning, Health care, Autonomous Navigation and Simulation. One of the major challenges in future frame generation is due to the high dimensional nature of visual data. In this work, we propose Mutual Information Predictive Auto-Encoder (MIPAE) framework, that reduces the task of predicting high dimensional video frames by factorising video representations into content and low dimensional pose latent variables that are easy to predict. A standard LSTM network is used to predict these low dimensional pose representations. Content and the predicted pose representations are decoded to generate future frames. Our approach leverages the temporal structure of the latent generative factors of a video and a novel mutual information loss to learn disentangled video representations. We also propose a metric based on mutual information gap (MIG) to quantitatively access the effectiveness of disentanglement on DSprites and MPI3D-real datasets. MIG scores corroborate with the visual superiority of frames predicted by MIPAE. We also compare our method quantitatively on evaluation metrics LPIPS, SSIM and PSNR.