Movie Genre Classification by Language Augmentation and Shot Sampling

TL;DR

This paper introduces Movie-CLIP, a novel approach for movie genre classification that leverages language elements and shot sampling to improve accuracy and efficiency, also extending to scene boundary detection.

Contribution

The paper proposes a new method combining language augmentation and shot sampling for more accurate and efficient movie genre classification and scene boundary detection.

Findings

Achieved 6-9% improvement in mean Average Precision over baselines.

Extended the approach to scene boundary detection with 1.1% AP improvement.

Demonstrated effectiveness on MovieNet and Condensed Movies datasets.

Abstract

Video-based movie genre classification has garnered considerable attention due to its various applications in recommendation systems. Prior work has typically addressed this task by adapting models from traditional video classification tasks, such as action recognition or event detection. However, these models often neglect language elements (e.g., narrations or conversations) present in videos, which can implicitly convey high-level semantics of movie genres, like storylines or background context. Additionally, existing approaches are primarily designed to encode the entire content of the input video, leading to inefficiencies in predicting movie genres. Movie genre prediction may require only a few shots to accurately determine the genres, rendering a comprehensive understanding of the entire video unnecessary. To address these challenges, we propose a Movie genre Classification method based on Language augmentatIon and shot samPling (Movie-CLIP). Movie-CLIP mainly consists of two parts: a language augmentation module to recognize language elements from the input audio, and a shot sampling module to select representative shots from the entire video. We evaluate our method on MovieNet and Condensed Movies datasets, achieving approximate 6-9% improvement in mean Average Precision (mAP) over the baselines. We also generalize Movie-CLIP to the scene boundary detection task, achieving 1.1% improvement in Average Precision (AP) over the state-of-the-art. We release our implementation at github.com/Zhongping-Zhang/Movie-CLIP.