Cinematic Behavior Transfer via NeRF-based Differentiable Filming

TL;DR

This paper presents a novel NeRF-based method for reverse filming behavior estimation and cinematic transfer, enabling precise manipulation of camera and character actions in 3D virtual environments with improved rendering and user control.

Contribution

It introduces a new pipeline combining NeRF and 3D engine workflows for cinematic behavior transfer, addressing limitations of existing methods in dynamic scenes and 3D pose estimation.

Findings

Enhanced camera trajectory optimization using NeRF

Successful transfer of shot types to new videos or 3D environments

Higher user satisfaction in rendering and control quality

Abstract

In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired. Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections, neglecting 3D statuses. To address these issues, we first introduce a reverse filming behavior estimation technique. It optimizes camera trajectories by leveraging NeRF as a differentiable renderer and refining SMPL tracks. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment. The incorporation of 3D engine workflow enables superior rendering and control abilities, which also achieves a higher rating in the user study.