Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation

TL;DR

This paper introduces a novel 3D-aware motion representation and a framework called Perception-as-Control for fine-grained, collaborative camera and object motion control in image animation, enabling more intuitive and consistent visual changes.

Contribution

It proposes a new 3D-aware motion representation and a unified framework for collaborative camera and object motion control in image animation tasks.

Findings

Outperforms existing methods in motion control quality

Supports diverse motion-related video synthesis tasks

Provides intuitive and consistent visual changes

Abstract

Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via various motion representations, while they still struggle to support collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user instructions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive and consistent visual changes. Then, our framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed approach. For more details and qualitative results, please refer to our anonymous project webpage: https://chen-yingjie.github.io/projects/Perception-as-Control.