Video Summarization by Learning from Unpaired Data

TL;DR

This paper introduces a novel method for video summarization that learns from unpaired data using adversarial training and diversity constraints, reducing the need for costly labeled datasets.

Contribution

It proposes a new unpaired learning framework for video summarization that does not require aligned training data, leveraging adversarial and diversity constraints.

Findings

Outperforms existing methods on benchmark datasets

Effectively learns from unpaired raw and summary videos

Reduces reliance on expensive labeled training data

Abstract

We consider the problem of video summarization. Given an input raw video, the goal is to select a small subset of key frames from the input video to create a shorter summary video that best describes the content of the original video. Most of the current state-of-the-art video summarization approaches use supervised learning and require labeled training data. Each training instance consists of a raw input video and its ground truth summary video curated by human annotators. However, it is very expensive and difficult to create such labeled training examples. To address this limitation, we propose a novel formulation to learn video summarization from unpaired data. We present an approach that learns to generate optimal video summaries using a set of raw videos ($V$) and a set of summary videos ($S$), where there exists no correspondence between $V$ and $S$. We argue that this type of data is much easier to collect. Our model aims to learn a mapping function $F : V \rightarrow S$ such that the distribution of resultant summary videos from $F(V)$ is similar to the distribution of $S$ with the help of an adversarial objective. In addition, we enforce a diversity constraint on $F(V)$ to ensure that the generated video summaries are visually diverse. Experimental results on two benchmark datasets indicate that our proposed approach significantly outperforms other alternative methods.