Multimodal Fusion and Coherence Modeling for Video Topic Segmentation

TL;DR

This paper advances video topic segmentation by developing multimodal fusion and coherence modeling techniques, including new pre-training and fine-tuning tasks, evaluated on English and Chinese educational videos.

Contribution

It introduces novel architectures for multimodal fusion, a multimodal contrastive learning pre-training, and new tasks for coherence modeling tailored to VTS, with extensive evaluation.

Findings

Superior performance on English lecture videos

Effective multimodal fusion with cross-attention and mixture of experts

Enhanced results on Chinese lecture dataset

Abstract

The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling. Specifically, (1) we enhance multimodal fusion by exploring different architectures using cross-attention and mixture of experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate the proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, we introduce a large-scale Chinese lecture video dataset to augment the existing English corpus, promoting further research in VTS. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines.