Language-Guided Graph Representation Learning for Video Summarization

TL;DR

This paper introduces a novel language-guided graph network for video summarization that captures global dependencies, supports multimodal customization, and significantly reduces inference time and model size.

Contribution

The paper proposes a new graph-based framework with language guidance for improved video summarization, addressing limitations of existing methods in capturing semantic dependencies and enabling user customization.

Findings

Outperforms existing methods on multiple benchmarks.

Reduces inference time by 87.8% and model parameters by 91.7%.

Effectively captures semantic and contextual relationships in videos.

Abstract

With the rapid growth of video content on social media, video summarization has become a crucial task in multimedia processing. However, existing methods face challenges in capturing global dependencies in video content and accommodating multimodal user customization. Moreover, temporal proximity between video frames does not always correspond to semantic proximity. To tackle these challenges, we propose a novel Language-guided Graph Representation Learning Network (LGRLN) for video summarization. Specifically, we introduce a video graph generator that converts video frames into a structured graph to preserve temporal order and contextual dependencies. By constructing forward, backward and undirected graphs, the video graph generator effectively preserves the sequentiality and contextual relationships of video content. We designed an intra-graph relational reasoning module with a dual-threshold graph convolution mechanism, which distinguishes semantically relevant frames from irrelevant ones between nodes. Additionally, our proposed language-guided cross-modal embedding module generates video summaries with specific textual descriptions. We model the summary generation output as a mixture of Bernoulli distribution and solve it with the EM algorithm. Experimental results show that our method outperforms existing approaches across multiple benchmarks. Moreover, we proposed LGRLN reduces inference time and model parameters by 87.8% and 91.7%, respectively. Our codes and pre-trained models are available at https://github.com/liwrui/LGRLN.