Unsupervised Transcript-assisted Video Summarization and Highlight Detection

TL;DR

This paper introduces an unsupervised, reinforcement learning-based multimodal framework that combines video frames and transcripts to improve video summarization and highlight detection, outperforming previous methods that use only visual data.

Contribution

It is the first to integrate transcripts and video frames within an RL framework for unsupervised video summarization and highlight detection, leveraging modality fusion.

Findings

Transcript integration improves summary quality.

Unsupervised training enables large-scale learning.

Method outperforms visual-only approaches.

Abstract

Video consumption is a key part of daily life, but watching entire videos can be tedious. To address this, researchers have explored video summarization and highlight detection to identify key video segments. While some works combine video frames and transcripts, and others tackle video summarization and highlight detection using Reinforcement Learning (RL), no existing work, to the best of our knowledge, integrates both modalities within an RL framework. In this paper, we propose a multimodal pipeline that leverages video frames and their corresponding transcripts to generate a more condensed version of the video and detect highlights using a modality fusion mechanism. The pipeline is trained within an RL framework, which rewards the model for generating diverse and representative summaries while ensuring the inclusion of video segments with meaningful transcript content. The unsupervised nature of the training allows for learning from large-scale unannotated datasets, overcoming the challenge posed by the limited size of existing annotated datasets. Our experiments show that using the transcript in video summarization and highlight detection achieves superior results compared to relying solely on the visual content of the video.