Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical Models

TL;DR

This paper introduces PHAV, a large synthetic human action video dataset generated via a novel procedural and physics-based approach using game engines, which improves action recognition models when combined with real data.

Contribution

The paper presents a new interpretable generative model for creating diverse, realistic, and physically plausible human action videos, including synthetic categories, and a multi-modal data generation pipeline.

Findings

Synthetic data boosts recognition performance on benchmarks.

PHAV dataset contains nearly 40,000 videos across 35 categories.

Synthetic videos outperform fine-tuned unsupervised generative models.

Abstract

Deep video action recognition models have been highly successful in recent years but require large quantities of manually annotated data, which are expensive and laborious to obtain. In this work, we investigate the generation of synthetic training data for video action recognition, as synthetic data have been successfully used to supervise models for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation, physics models and other components of modern game engines. With this model we generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for "Procedural Human Action Videos". PHAV contains a total of 39,982 videos, with more than 1,000 examples for each of 35 action categories. Our video generation approach is not limited to existing motion capture sequences: 14 of these 35 categories are procedurally defined synthetic actions. In addition, each video is represented with 6 different data modalities, including RGB, optical flow and pixel-level semantic labels. These modalities are generated almost simultaneously using the Multiple Render Targets feature of modern GPUs. In order to leverage PHAV, we introduce a deep multi-task (i.e. that considers action classes from multiple datasets) representation learning architecture that is able to simultaneously learn from synthetic and real video datasets, even when their action categories differ. Our experiments on the UCF-101 and HMDB-51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance. Our approach also significantly outperforms video representations produced by fine-tuning state-of-the-art unsupervised generative models of videos.