Learning to Cut by Watching Movies

TL;DR

This paper introduces a novel approach to computational video editing by learning to identify plausible cuts from a large dataset of edited videos, outperforming baselines and demonstrating real-world effectiveness.

Contribution

It presents a new task for learning video cut plausibility using contrastive learning on a large dataset, with a benchmark and human evaluation showing improved performance.

Findings

Model outperforms baselines significantly

Human studies favor the proposed model

Effective in real-world unedited videos

Abstract

Video content creation keeps growing at an incredible pace; yet, creating engaging stories remains challenging and requires non-trivial video editing expertise. Many video editing components are astonishingly hard to automate primarily due to the lack of raw video materials. This paper focuses on a new task for computational video editing, namely the task of raking cut plausibility. Our key idea is to leverage content that has already been edited to learn fine-grained audiovisual patterns that trigger cuts. To do this, we first collected a data source of more than 10K videos, from which we extract more than 255K cuts. We devise a model that learns to discriminate between real and artificial cuts via contrastive learning. We set up a new task and a set of baselines to benchmark video cut generation. We observe that our proposed model outperforms the baselines by large margins. To demonstrate our model in real-world applications, we conduct human studies in a collection of unedited videos. The results show that our model does a better job at cutting than random and alternative baselines.