Comprehensive Video Understanding: Video summarization with content-based video recommender design
Yudong Jiang, Kaixu Cui, Bo Peng, Changliang Xu

TL;DR
This paper introduces a content-based recommender system for video summarization using deep neural networks, incorporating scene and action recognition, multi-modal features, and data augmentation, achieving top results in a challenge.
Contribution
It formulates video summarization as a content-based recommendation problem and develops a scalable deep neural network with multi-task learning for improved performance.
Findings
Achieved first place in ICCV 2019 CoView Workshop Challenge
Demonstrated effectiveness of multi-modal features in summarization
Showed benefits of data augmentation and multi-task learning
Abstract
Video summarization aims to extract keyframes/shots from a long video. Previous methods mainly take diversity and representativeness of generated summaries as prior knowledge in algorithm design. In this paper, we formulate video summarization as a content-based recommender problem, which should distill the most useful content from a long video for users who suffer from information overload. A scalable deep neural network is proposed on predicting if one video segment is a useful segment for users by explicitly modelling both segment and video. Moreover, we accomplish scene and action recognition in untrimmed videos in order to find more correlations among different aspects of video understanding tasks. Also, our paper will discuss the effect of audio and visual features in summarization task. We also extend our work by data augmentation and multi-task learning for preventing the model…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
