A Multimodal Transformer for Live Streaming Highlight Prediction

TL;DR

This paper introduces a multimodal transformer model for live streaming highlight prediction that effectively handles real-time constraints, complex multimodal data, and limited annotations, outperforming existing methods.

Contribution

The paper presents a novel multimodal transformer with a Modality Temporal Alignment Module and Border-aware Pairwise Loss for improved live streaming highlight detection.

Findings

Model outperforms strong baselines on real-world and public datasets.

Effectively handles multimodal data including images, audio, and text.

Utilizes weak supervision from user feedback to enhance learning.

Abstract

Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic.