ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

TL;DR

ActCam is a zero-shot video generation method that jointly controls character motion and camera trajectory, improving scene consistency and motion fidelity without training.

Contribution

It introduces a novel staged conditioning approach for joint camera and motion control in video generation using pretrained diffusion models.

Findings

Outperforms pose-only control in camera adherence and motion fidelity.

Preferred in human evaluations, especially with large viewpoint changes.

Enables joint camera and motion control without training.

Abstract

For artistic applications, video generation requires fine-grained control over both performance and cinematography, i.e., the actor's motion and the camera trajectory. We present ActCam, a zero-shot method for video generation that jointly transfers character motion from a driving video into a new scene and enables per-frame control of intrinsic and extrinsic camera parameters. ActCam builds on any pretrained image-to-video diffusion model that accepts conditioning in terms of scene depth and character pose. Given a source video with a moving character and a target camera motion, ActCam generates pose and depth conditions that remain geometrically consistent across frames. We then run a single sampling process with a two-phase conditioning schedule: early denoising steps condition on both pose and sparse depth to enforce scene structure, after which depth is dropped and pose-only guidance refines high-frequency details without over-constraining the generation. We evaluate ActCam on multiple benchmarks spanning diverse character motions and challenging viewpoint changes. We find that, compared to pose-only control and other pose and camera methods, ActCam improves camera adherence and motion fidelity, and is preferred in human evaluations, especially under large viewpoint changes. Our results highlight that careful camera-consistent conditioning and staged guidance can enable strong joint camera and motion control without training. Project page: https://elkhomar.github.io/actcam/.