Movies2Scenes: Using Movie Metadata to Learn Scene Representation

TL;DR

This paper introduces a contrastive learning method leveraging movie metadata to develop a versatile scene representation that improves performance across classification, regression, and moderation tasks.

Contribution

It proposes a novel approach that uses movie metadata to guide contrastive learning, enabling effective scene representation without scene-level labels.

Findings

Outperforms state-of-the-art methods on multiple benchmark datasets.

Achieves 7.9% average improvement on classification tasks.

Demonstrates generalizability on video moderation tasks.

Abstract

Understanding scenes in movies is crucial for a variety of applications such as video moderation, search, and recommendation. However, labeling individual scenes is a time-consuming process. In contrast, movie level metadata (e.g., genre, synopsis, etc.) regularly gets produced as part of the film production process, and is therefore significantly more commonly available. In this work, we propose a novel contrastive learning approach that uses movie metadata to learn a general-purpose scene representation. Specifically, we use movie metadata to define a measure of movie similarity, and use it during contrastive learning to limit our search for positive scene-pairs to only the movies that are considered similar to each other. Our learned scene representation consistently outperforms existing state-of-the-art methods on a diverse set of tasks evaluated using multiple benchmark datasets. Notably, our learned representation offers an average improvement of 7.9% on the seven classification tasks and 9.7% improvement on the two regression tasks in LVU dataset. Furthermore, using a newly collected movie dataset, we present comparative results of our scene representation on a set of video moderation tasks to demonstrate its generalizability on previously less explored tasks.