Jointly Trained Image and Video Generation using Residual Vectors

TL;DR

This paper introduces a joint training method for image and video generation models using residual vectors to encode temporal changes, improving sample quality and diversity.

Contribution

It presents a novel approach that jointly trains image and video generators with residual vectors, enabling better temporal modeling and shared information across frames.

Findings

Enhanced sample quality and diversity in generated images and videos

Compatibility with pre-training on mixed datasets of images and videos

Effective exploitation of feature similarity across frames

Abstract

In this work, we propose a modeling technique for jointly training image and video generation models by simultaneously learning to map latent variables with a fixed prior onto real images and interpolate over images to generate videos. The proposed approach models the variations in representations using residual vectors encoding the change at each time step over a summary vector for the entire video. We utilize the technique to jointly train an image generation model with a fixed prior along with a video generation model lacking constraints such as disentanglement. The joint training enables the image generator to exploit temporal information while the video generation model learns to flexibly share information across frames. Moreover, experimental results verify our approach's compatibility with pre-training on videos or images and training on datasets containing a mixture of both. A comprehensive set of quantitative and qualitative evaluations reveal the improvements in sample quality and diversity over both video generation and image generation baselines. We further demonstrate the technique's capabilities of exploiting similarity in features across frames by applying it to a model based on decomposing the video into motion and content. The proposed model allows minor variations in content across frames while maintaining the temporal dependence through latent vectors encoding the pose or motion features.