Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography

TL;DR

This paper introduces a deep learning framework for automatic camera control in virtual cinematography that synchronizes camera movements with actor actions and emotions to produce immersive, high-quality cinematic videos.

Contribution

It presents a novel actor-camera synchronization method considering aesthetics, physical action, and emotional state, using a GAN-based model and self-supervised adjustments.

Findings

Produces immersive cinematic videos with high quality

Effectively synchronizes camera with actor’s actions and emotions

Outperforms existing methods in qualitative and quantitative evaluations

Abstract

User-generated cinematic creations are gaining popularity as our daily entertainment, yet it is a challenge to master cinematography for producing immersive contents. Many existing automatic methods focus on roughly controlling predefined shot types or movement patterns, which struggle to engage viewers with the circumstances of the actor. Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects, considering frame aesthetics, spatial action, and emotional status in the 3D virtual stage. Following rule-of-thirds, our framework first modifies the initial camera placement to position the actor aesthetically. This adjustment is facilitated by a self-supervised adjustor that analyzes frame composition via camera projection. We then design a GAN model that can adversarially synthesize fine-grained camera movement based on the physical action and psychological state of the actor, using an encoder-decoder generator to map kinematics and emotional variables into camera trajectories. Moreover, we incorporate a regularizer to align the generated stylistic variances with specific emotional categories and intensities. The experimental results show that our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively. Live examples can be found in the supplementary video.