Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders

TL;DR

This paper introduces an unsupervised method using robust recurrent auto-encoders with a novel loss function to automatically extract video highlights from user-edited videos, avoiding the need for labeled data.

Contribution

It proposes a new unsupervised approach employing a robust recurrent auto-encoder with a shrinking exponential loss to identify significant sub-events in videos.

Findings

Effective highlight extraction demonstrated on social media videos

Robustness to noisy web-crawled data confirmed

Outperforms some existing unsupervised methods

Abstract

With the growing popularity of short-form video sharing platforms such as \em{Instagram} and \em{Vine}, there has been an increasing need for techniques that automatically extract highlights from video. Whereas prior works have approached this problem with heuristic rules or supervised learning, we present an unsupervised learning approach that takes advantage of the abundance of user-edited videos on social media websites such as YouTube. Based on the idea that the most significant sub-events within a video class are commonly present among edited videos while less interesting ones appear less frequently, we identify the significant sub-events via a robust recurrent auto-encoder trained on a collection of user-edited videos queried for each particular class of interest. The auto-encoder is trained using a proposed shrinking exponential loss function that makes it robust to noise in the web-crawled training data, and is configured with bidirectional long short term memory (LSTM)~\cite{LSTM:97} cells to better model the temporal structure of highlight segments. Different from supervised techniques, our method can infer highlights using only a set of downloaded edited videos, without also needing their pre-edited counterparts which are rarely available online. Extensive experiments indicate the promise of our proposed solution in this challenging unsupervised settin