CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation

TL;DR

CineTechBench is a new benchmark with expert annotations designed to evaluate and improve multimodal models' ability to understand and generate cinematographic techniques in film, covering key visual aspects with over 600 images and 120 clips.

Contribution

It introduces a comprehensive, expert-annotated benchmark for cinematography, enabling systematic evaluation of models' understanding and generation of film techniques.

Findings

Current models show limitations in understanding cinematographic nuances.

The benchmark reveals gaps in models' ability to generate cinema-quality camera movements.

Insights guide future research in film-related AI applications.

Abstract

Cinematography is a cornerstone of film production and appreciation, shaping mood, emotion, and narrative through visual elements such as camera movement, shot composition, and lighting. Despite recent progress in multimodal large language models (MLLMs) and video generation models, the capacity of current models to grasp and reproduce cinematographic techniques remains largely uncharted, hindered by the scarcity of expert-annotated data. To bridge this gap, we present CineTechBench, a pioneering benchmark founded on precise, manual annotation by seasoned cinematography experts across key cinematography dimensions. Our benchmark covers seven essential aspects-shot scale, shot angle, composition, camera movement, lighting, color, and focal length-and includes over 600 annotated movie images and 120 movie clips with clear cinematographic techniques. For the understanding task, we design question answer pairs and annotated descriptions to assess MLLMs' ability to interpret and explain cinematographic techniques. For the generation task, we assess advanced video generation models on their capacity to reconstruct cinema-quality camera movements given conditions such as textual prompts or keyframes. We conduct a large-scale evaluation on 15+ MLLMs and 5+ video generation models. Our results offer insights into the limitations of current models and future directions for cinematography understanding and generation in automatically film production and appreciation. The code and benchmark can be accessed at https://github.com/PRIS-CV/CineTechBench.