E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awareness

TL;DR

This paper introduces the E.T. dataset for character-aware camera trajectories and a diffusion-based model, DIRECTOR, that generates camera paths from textual descriptions, advancing accessible cinematography.

Contribution

The paper presents the first dataset of character-aware camera trajectories and a novel diffusion-based method for text-to-camera-trajectory generation.

Findings

The E.T. dataset enables new research in camera trajectory generation.

DIRECTOR effectively generates complex camera paths from textual captions.

Evaluation shows improved accuracy with the CLaTr embedding.

Abstract

Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, in particular camera placement and movement over time. Crafting compelling camera trajectories remains a complex iterative process, even for skilful artists. To tackle this, in this paper, we propose a dataset called the Exceptional Trajectories (E.T.) with camera trajectories along with character information and textual captions encompassing descriptions of both camera and character. To our knowledge, this is the first dataset of its kind. To show the potential applications of the E.T. dataset, we propose a diffusion-based approach, named DIRECTOR, which generates complex camera trajectories from textual captions that describe the relation and synchronisation between the camera and characters. To ensure robust and accurate evaluations, we train on the E.T. dataset CLaTr, a Contrastive Language-Trajectory embedding for evaluation metrics. We posit that our proposed dataset and method significantly advance the democratization of cinematography, making it more accessible to common users.