Conditional Temporal Variational AutoEncoder for Action Video Prediction

TL;DR

This paper introduces ACT-VAE, a deep generative model that predicts pose sequences and synthesizes realistic action videos from a single image, effectively modeling motion and diversity with improved accuracy.

Contribution

The paper presents a novel temporal variational autoencoder framework that enhances action sequence prediction and diversity modeling, integrated with a pose-to-image network for video synthesis.

Findings

Outperforms state-of-the-art in pose prediction accuracy

Successfully synthesizes realistic action video sequences

Maintains diversity in generated action sequences

Abstract

To synthesize a realistic action sequence based on a single human image, it is crucial to model both motion patterns and diversity in the action video. This paper proposes an Action Conditional Temporal Variational AutoEncoder (ACT-VAE) to improve motion prediction accuracy and capture movement diversity. ACT-VAE predicts pose sequences for an action clips from a single input image. It is implemented as a deep generative model that maintains temporal coherence according to the action category with a novel temporal modeling on latent space. Further, ACT-VAE is a general action sequence prediction framework. When connected with a plug-and-play Pose-to-Image (P2I) network, ACT-VAE can synthesize image sequences. Extensive experiments bear out our approach can predict accurate pose and synthesize realistic image sequences, surpassing state-of-the-art approaches. Compared to existing methods, ACT-VAE improves model accuracy and preserves diversity.