Video Summarization with Long Short-term Memory

TL;DR

This paper introduces a supervised learning approach using Long Short-Term Memory networks for video summarization, effectively modeling sequential dependencies to select keyframes or subshots, achieving state-of-the-art results on benchmark datasets.

Contribution

The paper presents a novel LSTM-based structured prediction model for video summarization and introduces domain adaptation techniques to leverage heterogeneous annotated datasets.

Findings

Achieved state-of-the-art results on benchmark datasets.

Modeling sequential structures is crucial for effective video summarization.

Domain adaptation improves performance by reducing dataset discrepancies.

Abstract

We propose a novel supervised learning technique for summarizing videos by automatically selecting keyframes or key subshots. Casting the problem as a structured prediction problem on sequential data, our main idea is to use Long Short-Term Memory (LSTM), a special type of recurrent neural networks to model the variable-range dependencies entailed in the task of video summarization. Our learning models attain the state-of-the-art results on two benchmark video datasets. Detailed analysis justifies the design of the models. In particular, we show that it is crucial to take into consideration the sequential structures in videos and model them. Besides advances in modeling techniques, we introduce techniques to address the need of a large number of annotated data for training complex learning models. There, our main idea is to exploit the existence of auxiliary annotated video datasets, albeit heterogeneous in visual styles and contents. Specifically, we show domain adaptation techniques can improve summarization by reducing the discrepancies in statistical properties across those datasets.