DTVNet+: A High-Resolution Scenic Dataset for Dynamic Time-lapse Video Generation

TL;DR

This paper introduces DTVNet+, a framework for generating high-resolution, diverse time-lapse videos from a single landscape image using motion vectors, and presents a new scenic dataset for evaluation.

Contribution

The paper proposes DTVNet+, a novel end-to-end model combining optical flow encoding and dynamic video generation with adaptive normalization, and introduces the Quick-Sky-Time dataset for benchmarking high-quality scenic video synthesis.

Findings

DTVNet+ outperforms existing methods in video quality and diversity.

The new dataset enables effective evaluation of scenic time-lapse video generation.

Adaptive instance normalization improves control over object motion in generated videos.

Abstract

This paper presents a novel end-to-end dynamic time-lapse video generation framework, named DTVNet, to generate diversified time-lapse videos from a single landscape image conditioned on normalized motion vectors. The proposed DTVNet consists of two submodules: \emph{Optical Flow Encoder} (OFE) and \emph{Dynamic Video Generator} (DVG). The OFE maps a sequence of optical flow maps to a \emph{normalized motion vector} that encodes the motion information of the generated video. The DVG contains motion and content streams to learn from the motion vector and the single landscape image. Besides, it contains an encoder to learn shared content features and a decoder to construct video frames with corresponding motion. Specifically, the \emph{motion stream} introduces multiple \emph{adaptive instance normalization} (AdaIN) layers to integrate multi-level motion information for controlling the object motion. In the testing stage, videos with the same content but various motion information can be generated by different \emph{normalized motion vectors} based on only one input image. Also, we propose a high-resolution scenic time-lapse video dataset, named Quick-Sky-Time, to evaluate different approaches, which can be viewed as a new benchmark for high-quality scenic image and video generation tasks. We further conduct experiments on Sky Time-lapse, Beach, and Quick-Sky-Time datasets. The results demonstrate the superiority of our approach over state-of-the-art methods for generating high-quality and various dynamic videos.