A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model

TL;DR

This paper introduces a novel two-stage automatic video editing approach that uses a pre-trained vision-language model for context extraction and reinforcement learning to improve editing decisions, applicable to diverse video content.

Contribution

The work presents a general editing framework leveraging VLM for context and RL for decision-making, addressing the limitations of scene-specific editing systems.

Findings

Effective context representation with VLM improves editing relevance.

RL-based framework enhances editing quality and decision accuracy.

Method demonstrates superior performance on a real movie dataset.

Abstract

In this era of videos, automatic video editing techniques attract more and more attention from industry and academia since they can reduce workloads and lower the requirements for human editors. Existing automatic editing systems are mainly scene- or event-specific, e.g., soccer game broadcasting, yet the automatic systems for general editing, e.g., movie or vlog editing which covers various scenes and events, were rarely studied before, and converting the event-driven editing method to a general scene is nontrivial. In this paper, we propose a two-stage scheme for general editing. Firstly, unlike previous works that extract scene-specific features, we leverage the pre-trained Vision-Language Model (VLM) to extract the editing-relevant representations as editing context. Moreover, to close the gap between the professional-looking videos and the automatic productions generated with simple guidelines, we propose a Reinforcement Learning (RL)-based editing framework to formulate the editing problem and train the virtual editor to make better sequential editing decisions. Finally, we evaluate the proposed method on a more general editing task with a real movie dataset. Experimental results demonstrate the effectiveness and benefits of the proposed context representation and the learning ability of our RL-based editing framework.