Comprehensive Video Understanding: Video summarization with content-based video recommender design

TL;DR

This paper introduces a content-based recommender system for video summarization using deep neural networks, incorporating scene and action recognition, multi-modal features, and data augmentation, achieving top results in a challenge.

Contribution

It formulates video summarization as a content-based recommendation problem and develops a scalable deep neural network with multi-task learning for improved performance.

Findings

Achieved first place in ICCV 2019 CoView Workshop Challenge

Demonstrated effectiveness of multi-modal features in summarization

Showed benefits of data augmentation and multi-task learning

Abstract

Video summarization aims to extract keyframes/shots from a long video. Previous methods mainly take diversity and representativeness of generated summaries as prior knowledge in algorithm design. In this paper, we formulate video summarization as a content-based recommender problem, which should distill the most useful content from a long video for users who suffer from information overload. A scalable deep neural network is proposed on predicting if one video segment is a useful segment for users by explicitly modelling both segment and video. Moreover, we accomplish scene and action recognition in untrimmed videos in order to find more correlations among different aspects of video understanding tasks. Also, our paper will discuss the effect of audio and visual features in summarization task. We also extend our work by data augmentation and multi-task learning for preventing the model from early-stage overfitting. The final results of our model win the first place in ICCV 2019 CoView Workshop Challenge Track.