SSGVS: Semantic Scene Graph-to-Video Synthesis

TL;DR

This paper introduces SSGVS, a novel framework that uses semantic scene graphs to guide video synthesis, enabling explicit temporal control and improved generation quality.

Contribution

The paper proposes a new video synthesis method leveraging semantic scene graphs and a specialized encoder, enhancing temporal guidance and prediction for unlabeled frames.

Findings

Outperforms existing models on Action Genome dataset

Effectively encodes and predicts scene graph representations

Demonstrates the importance of scene graphs in video synthesis

Abstract

As a natural extension of the image synthesis task, video synthesis has attracted a lot of interest recently. Many image synthesis works utilize class labels or text as guidance. However, neither labels nor text can provide explicit temporal guidance, such as when an action starts or ends. To overcome this limitation, we introduce semantic video scene graphs as input for video synthesis, as they represent the spatial and temporal relationships between objects in the scene. Since video scene graphs are usually temporally discrete annotations, we propose a video scene graph (VSG) encoder that not only encodes the existing video scene graphs but also predicts the graph representations for unlabeled frames. The VSG encoder is pre-trained with different contrastive multi-modal losses. A semantic scene graph-to-video synthesis framework (SSGVS), based on the pre-trained VSG encoder, VQ-VAE, and auto-regressive Transformer, is proposed to synthesize a video given an initial scene image and a non-fixed number of semantic scene graphs. We evaluate SSGVS and other state-of-the-art video synthesis models on the Action Genome dataset and demonstrate the positive significance of video scene graphs in video synthesis. The source code will be released.