Learning Fine-Grained Motion Embedding for Landscape Animation

TL;DR

This paper introduces FGLA, a novel model for landscape animation that learns fine-grained motion embeddings to generate realistic time-lapse videos from single images, significantly improving motion accuracy and visual quality.

Contribution

It proposes a new motion embedding approach and a large high-resolution dataset, advancing the realism and diversity of landscape animation from static images.

Findings

19% improvement on LIPIS metric

5.6% improvement on FVD metric

Outperforms existing methods in user studies

Abstract

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.