ESA: Energy-Based Shot Assembly Optimization for Automatic Video Editing

TL;DR

This paper introduces ESA, an energy-based optimization framework for automatic video shot assembly that aligns sequences with artistic styles and narrative requirements, enhancing automated editing with style learning.

Contribution

The paper presents a novel energy-based model that learns from reference videos to automate shot assembly, capturing artistic and narrative styles in video editing.

Findings

Successfully automates shot sequencing aligning with reference styles

Learns and replicates artistic styles from reference videos

Enables users with no editing experience to produce compelling videos

Abstract

Shot assembly is a crucial step in film production and video editing, involving the sequencing and arrangement of shots to construct a narrative, convey information, or evoke emotions. Traditionally, this process has been manually executed by experienced editors. While current intelligent video editing technologies can handle some automated video editing tasks, they often fail to capture the creator's unique artistic expression in shot assembly. To address this challenge, we propose an energy-based optimization method for video shot assembly. Specifically, we first perform visual-semantic matching between the script generated by a large language model and a video library to obtain subsets of candidate shots aligned with the script semantics. Next, we segment and label the shots from reference videos, extracting attributes such as shot size, camera motion, and semantics. We then employ energy-based models to learn from these attributes, scoring candidate shot sequences based on their alignment with reference styles. Finally, we achieve shot assembly optimization by combining multiple syntax rules, producing videos that align with the assembly style of the reference videos. Our method not only automates the arrangement and combination of independent shots according to specific logic, narrative requirements, or artistic styles but also learns the assembly style of reference videos, creating a coherent visual sequence or holistic visual expression. With our system, even users with no prior video editing experience can create visually compelling videos. Project page: https://sobeymil.github.io/esa.com