SketchAnimator: Animate Sketch via Motion Customization of Text-to-Video Diffusion Models

TL;DR

SketchAnimator is a novel model that animates sketches by learning appearance and motion from reference videos, enabling easy, customizable sketch animation without extensive manual effort.

Contribution

The paper introduces SketchAnimator, a new approach that combines appearance learning, motion learning, and video prior distillation for one-shot sketch animation.

Findings

Produces sketch videos that retain original appearance and mimic reference motion.

Effective one-shot motion customization for sketch animation.

Outperforms alternative methods in generating desired sketch videos.

Abstract

Sketching is a uniquely human tool for expressing ideas and creativity. The animation of sketches infuses life into these static drawings, opening a new dimension for designers. Animating sketches is a time-consuming process that demands professional skills and extensive experience, often proving daunting for amateurs. In this paper, we propose a novel sketch animation model SketchAnimator, which enables adding creative motion to a given sketch, like "a jumping car''. Namely, given an input sketch and a reference video, we divide the sketch animation into three stages: Appearance Learning, Motion Learning and Video Prior Distillation. In stages 1 and 2, we utilize LoRA to integrate sketch appearance information and motion dynamics from the reference video into the pre-trained T2V model. In the third stage, we utilize Score Distillation Sampling (SDS) to update the parameters of the Bezier curves in each sketch frame according to the acquired motion information. Consequently, our model produces a sketch video that not only retains the original appearance of the sketch but also mirrors the dynamic movements of the reference video. We compare our method with alternative approaches and demonstrate that it generates the desired sketch video under the challenge of one-shot motion customization.