DeepLandscape: Adversarial Modeling of Landscape Video

TL;DR

DeepLandscape introduces an extended StyleGAN-based model capable of generating realistic landscape videos with dynamic elements and time-of-day changes, enabling reenactment and manipulation of static images.

Contribution

The paper presents a novel architecture extending StyleGAN for landscape video synthesis, including a new inversion method for realistic image manipulation.

Findings

Produces more compelling landscape animations than prior methods.

Enables realistic reenactment of static landscape images.

Demonstrates effective modeling of dynamic scene changes.

Abstract

We build a new model of landscape videos that can be trained on a mixture of static landscape images as well as landscape animations. Our architecture extends StyleGAN model by augmenting it with parts that allow to model dynamic changes in a scene. Once trained, our model can be used to generate realistic time-lapse landscape videos with moving objects and time-of-the-day changes. Furthermore, by fitting the learned models to a static landscape image, the latter can be reenacted in a realistic way. We propose simple but necessary modifications to StyleGAN inversion procedure, which lead to in-domain latent codes and allow to manipulate real images. Quantitative comparisons and user studies suggest that our model produces more compelling animations of given photographs than previously proposed methods. The results of our approach including comparisons with prior art can be seen in supplementary materials and on the project page https://saic-mdal.github.io/deep-landscape