ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

TL;DR

This paper introduces ConCLVD, a diffusion-based model for controllable Chinese landscape video generation, utilizing a new dataset and techniques to produce dynamic, high-quality landscape videos that preserve artistic style.

Contribution

The paper presents a novel dataset and a diffusion model with a dual attention motion module and contrastive learning, enabling controllable and realistic Chinese landscape video synthesis.

Findings

Effective preservation of landscape style and dynamics

High-quality, smooth landscape videos generated

Advancement in artistic video diffusion methods

Abstract

Chinese landscape painting is a gem of Chinese cultural and artistic heritage that showcases the splendor of nature through the deep observations and imaginations of its painters. Limited by traditional techniques, these artworks were confined to static imagery in ancient times, leaving the dynamism of landscapes and the subtleties of artistic sentiment to the viewer's imagination. Recently, emerging text-to-video (T2V) diffusion methods have shown significant promise in video generation, providing hope for the creation of dynamic Chinese landscape paintings. However, challenges such as the lack of specific datasets, the intricacy of artistic styles, and the creation of extensive, high-quality videos pose difficulties for these models in generating Chinese landscape painting videos. In this paper, we propose CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we present a motion module featuring a dual attention mechanism to capture the dynamic transformations of landscape imageries, alongside a noise adapter to leverage unsupervised contrastive learning in the latent space. Following the generation of keyframes, we employ optical flow for frame interpolation to enhance video smoothness. Our method not only retains the essence of the landscape painting imageries but also achieves dynamic transitions, significantly advancing the field of artistic video generation. The source code and dataset are available at https://anonymous.4open.science/r/ConCLVD-EFE3.