FullTransNet: Full Transformer with Local-Global Attention for Video Summarization

TL;DR

FullTransNet introduces a full transformer architecture with local-global sparse attention for video summarization, improving modeling of dependencies and efficiency over existing methods.

Contribution

It proposes a novel full transformer with local-global attention for video summarization, addressing limitations of previous CNN and encoder-only transformer approaches.

Findings

Achieves state-of-the-art F-scores on SumMe and TVSum datasets.

Reduces computational costs compared to traditional full attention models.

Outperforms second-best methods by small margins, confirming effectiveness.

Abstract

Video summarization aims to generate a compact, informative, and representative synopsis of raw videos, which is crucial for browsing, analyzing, and understanding video content. Dominant approaches in video summarization primarily rely on recurrent or convolutional neural networks, and more recently on encoder-only transformer architectures. However, these methods typically suffer from several limitations in parallelism, modeling long-range dependencies, and providing explicit generative capabilities. To address these issues, we propose a transformer-like architecture named FullTransNet with two-fold ideas. First, it uses a full transformer with an encoder-decoder structure as an alternative architecture for video summarization. As the full transformer is specifically designed for sequence transduction tasks, its direct application to video summarization is both intuitive and effective. Second, it replaces the standard full attention mechanism with a combination of local and global sparse attention, enabling the model to capture long-range dependencies while significantly reducing computational costs. This local-global sparse attention is applied exclusively at the encoder side, where the majority of computations occur, further enhancing efficiency. Extensive experiments on two widely used benchmark datasets, SumMe and TVSum, demonstrate that our model achieves F-scores of 54.4% and 63.9%, respectively, while maintaining relatively low computational and memory requirements. These results surpass the second-best performing methods by 0.1% and 0.3%, respectively, verifying the effectiveness and efficiency of FullTransNet.