CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion

TL;DR

CinePreGen is a system that enhances video previsualization by integrating engine-powered diffusion with intuitive camera controls, enabling more precise and realistic cinematic camera movements in AI-generated videos.

Contribution

It introduces a novel camera and storyboard interface combined with an AI rendering workflow to improve control, consistency, and realism in AI-based video previsualization.

Findings

Reduces development complexity and challenges.

Meets user needs for extensive control and iteration.

Outperforms existing workflows in cinematic camera movement.

Abstract

With advancements in video generative AI models (e.g., SORA), creators are increasingly using these techniques to enhance video previsualization. However, they face challenges with incomplete and mismatched AI workflows. Existing methods mainly rely on text descriptions and struggle with camera placement, a key component of previsualization. To address these issues, we introduce CinePreGen, a visual previsualization system enhanced with engine-powered diffusion. It features a novel camera and storyboard interface that offers dynamic control, from global to local camera adjustments. This is combined with a user-friendly AI rendering workflow, which aims to achieve consistent results through multi-masked IP-Adapter and engine simulation guidelines. In our comprehensive evaluation study, we demonstrate that our system reduces development viscosity (i.e., the complexity and challenges in the development process), meets users' needs for extensive control and iteration in the design process, and outperforms other AI video production workflows in cinematic camera movement, as shown by our experiments and a within-subjects user study. With its intuitive camera controls and realistic rendering of camera motion, CinePreGen shows great potential for improving video production for both individual creators and industry professionals.