Video Summarization through Reinforcement Learning with a 3D Spatio-Temporal U-Net

TL;DR

This paper presents a novel reinforcement learning framework using a 3D spatio-temporal U-Net for efficient video summarization, capable of operating in supervised and unsupervised modes, and applicable to medical videos.

Contribution

Introduction of the 3DST-UNet-RL framework that combines 3D CNN encoding with reinforcement learning for improved video summarization performance.

Findings

Effective in both supervised and unsupervised modes.

Outperforms baseline methods on standard benchmarks.

Applicable to medical video summarization, reducing storage and improving review efficiency.

Abstract

Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatio-temporal U-Net is used to efficiently encode spatio-temporal information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatio-temporal latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatio-temporal CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information