DreamCinema: Cinematic Transfer with Free Camera and 3D Character

TL;DR

DreamCinema introduces a novel framework for 3D film creation that combines user-specified cinematic elements with generative models, enabling high-quality, immersive videos with free camera movement and consistent 3D characters.

Contribution

The paper presents a new 3D film creation framework that decomposes cinematic elements and integrates them using structure-guided animation and environment-aware refinement.

Findings

Effective generation of high-quality 3D films with free camera movement

Consistent 3D character modeling across frames

Improved cinematic quality through element decomposition and refinement

Abstract

We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on video generation suggests a promising avenue for controllable film creation in pixel space using Diffusion models. However, the reliance on overly verbose prompts and insufficient focus on cinematic elements (e.g., camera movement) results in videos that lack cinematic quality. Furthermore, the absence of 3D modeling often leads to failures in video generation, such as inconsistent character models at different frames, ultimately hindering the immersive experience for viewers. In this paper, we propose a new framework for film creation, Dream-Cinema, which is designed for user-friendly, 3D space-based film creation with generative models. Specifically, we decompose 3D film creation into four key elements: 3D character, driven motion, camera movement, and environment. We extract the latter three elements from user-specified film shots and generate the 3D character using a generative model based on a provided image. To seamlessly recombine these elements and ensure smooth film creation, we propose structure-guided character animation, shape-aware camera movement optimization, and environment-aware generative refinement. Extensive experiments demonstrate the effectiveness of our method in generating high-quality films with free camera and 3D characters.