MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generation

TL;DR

MotionCanvas introduces a user-controllable image-to-video generation method that enables intuitive design of camera and object motions, enhancing cinematic shot creation without extensive 3D data.

Contribution

It presents a novel approach integrating user-driven controls into I2V models for scene-aware motion editing, bridging classical graphics and modern diffusion techniques.

Findings

Effective 3D-aware motion control without costly 3D data

Intuitive depiction of scene-space motion intentions

Versatile application across real-world content and editing scenarios

Abstract

This paper presents a method that allows users to design cinematic video shots in the context of image-to-video generation. Shot design, a critical aspect of filmmaking, involves meticulously planning both camera movements and object motions in a scene. However, enabling intuitive shot design in modern image-to-video generation systems presents two main challenges: first, effectively capturing user intentions on the motion design, where both camera movements and scene-space object motions must be specified jointly; and second, representing motion information that can be effectively utilized by a video diffusion model to synthesize the image animations. To address these challenges, we introduce MotionCanvas, a method that integrates user-driven controls into image-to-video (I2V) generation models, allowing users to control both object and camera motions in a scene-aware manner. By connecting insights from classical computer graphics and contemporary video generation techniques, we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis without requiring costly 3D-related training data. MotionCanvas enables users to intuitively depict scene-space motion intentions, and translates them into spatiotemporal motion-conditioning signals for video diffusion models. We demonstrate the effectiveness of our method on a wide range of real-world image content and shot-design scenarios, highlighting its potential to enhance the creative workflows in digital content creation and adapt to various image and video editing applications.