Text-Guided Synthesis of Eulerian Cinemagraphs

TL;DR

This paper presents Text2Cinemagraph, an automated method that generates artistic cinemagraphs from text prompts by synthesizing image pairs and transferring realistic motion to artistic images, improving consistency and artistic control.

Contribution

It introduces a novel approach of creating image twins from text prompts, enabling effective motion transfer for artistic and natural scenes in cinemagraph synthesis.

Findings

Outperforms existing methods in natural and artistic cinemagraphs

Achieves higher consistency and realism validated by metrics and user studies

Enables animation of paintings and control of motion directions

Abstract

We introduce Text2Cinemagraph, a fully automated method for creating cinemagraphs from text descriptions - an especially challenging task when prompts feature imaginary elements and artistic styles, given the complexity of interpreting the semantics and motions of these images. We focus on cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds, which exhibit continuous motion and repetitive textures. Existing single-image animation methods fall short on artistic inputs, and recent text-based video methods frequently introduce temporal inconsistencies, struggling to keep certain regions static. To address these challenges, we propose an idea of synthesizing image twins from a single text prompt - a pair of an artistic image and its pixel-aligned corresponding natural-looking twin. While the artistic image depicts the style and appearance detailed in our text prompt, the realistic counterpart greatly simplifies layout and motion analysis. Leveraging existing natural image and video datasets, we can accurately segment the realistic image and predict plausible motion given the semantic information. The predicted motion can then be transferred to the artistic image to create the final cinemagraph. Our method outperforms existing approaches in creating cinemagraphs for natural landscapes as well as artistic and other-worldly scenes, as validated by automated metrics and user studies. Finally, we demonstrate two extensions: animating existing paintings and controlling motion directions using text.