Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding

TL;DR

This paper introduces M-SYMON, a large-scale multilingual dataset of movie summaries with fine-grained video-text annotations, enabling improved multilingual story understanding and alignment in videos across seven languages.

Contribution

The creation of the M-SYMON dataset with manual annotations for multilingual video-text alignment and benchmarking baseline approaches for future research.

Findings

Training on M-SYMON data outperforms state-of-the-art methods.

Multilingual training strategies show different performances in intra- and cross-lingual settings.

The dataset facilitates research in multilingual video-story understanding.

Abstract

Story video-text alignment, a core task in computational story understanding, aims to align video clips with corresponding sentences in their descriptions. However, progress on the task has been held back by the scarcity of manually annotated video-text correspondence and the heavy concentration on English narrations of Hollywood movies. To address these issues, in this paper, we construct a large-scale multilingual video story dataset named Multilingual Synopses of Movie Narratives (M-SYMON), containing 13,166 movie summary videos from 7 languages, as well as manual annotation of fine-grained video-text correspondences for 101.5 hours of video. Training on the human annotated data from SyMoN outperforms the SOTA methods by 15.7 and 16.2 percentage points on Clip Accuracy and Sentence IoU scores, respectively, demonstrating the effectiveness of the annotations. As benchmarks for future research, we create 6 baseline approaches with different multilingual training strategies, compare their performance in both intra-lingual and cross-lingual setups, exemplifying the challenges of multilingual video-text alignment. The dataset is released at: https://github.com/insundaycathy/M-SyMoN