Sketch-Guided Stylized Landscape Cinemagraph Synthesis

TL;DR

This paper introduces Sketch2Cinemagraph, a novel framework that uses sketches and text prompts to generate stylized, controllable cinemagraphs with realistic motion, enhancing customization and aesthetic appeal.

Contribution

The paper presents a new sketch-guided method for generating stylized cinemagraphs with controllable motion fields, combining diffusion models and object detection for improved customization.

Findings

Effective control of motion regions via sketches

High-quality stylized cinemagraphs with continuous flow

Outperforms state-of-the-art methods in qualitative and quantitative tests

Abstract

Designing stylized cinemagraphs is challenging due to the difficulty in customizing complex and expressive flow elements. To achieve intuitive and detailed control of the generated cinemagraphs, sketches provide a feasible solution to convey personalized design requirements beyond text inputs. In this paper, we propose Sketch2Cinemagraph, a sketch-guided framework that enables the conditional generation of stylized cinemagraphs from freehand sketches. Sketch2Cinemagraph adopts text prompts for initial landscape generation and provides sketch controls for both spatial and motion cues. The latent diffusion model first generates target stylized landscape images along with realistic versions. Then, a pre-trained object detection model obtains masks for the flow regions. We propose a latent motion diffusion model to estimate motion field in fluid regions of the generated landscape images. The input motion sketches serve as the conditions to control the generated motion fields in the masked fluid regions with the prompt. To synthesize cinemagraph frames, the pixels within fluid regions are warped to target locations at each timestep using a U-Net based frame generator. The results verified that Sketch2Cinemagraph can generate aesthetically appealing stylized cinemagraphs with continuous temporal flow from sketch inputs. We showcase the advantages of Sketch2Cinemagraph through qualitative and quantitative comparisons against the state-of-the-art approaches.