Unsupervised Semantic Parsing of Video Collections

TL;DR

This paper introduces an unsupervised method for parsing videos into semantic steps, creating a storyline with textual descriptions using visual and language cues, demonstrated on complex YouTube videos.

Contribution

It presents a novel unsupervised approach combining visual and language cues to generate semantic storylines and descriptions from videos.

Findings

Achieved high-quality semantic parsing of complex videos

Generated accurate textual descriptions for video segments

Demonstrated effectiveness on large YouTube video dataset

Abstract

Human communication typically has an underlying structure. This is reflected in the fact that in many user generated videos, a starting point, ending, and certain objective steps between these two can be identified. In this paper, we propose a method for parsing a video into such semantic steps in an unsupervised way. The proposed method is capable of providing a semantic "storyline" of the video composed of its objective steps. We accomplish this using both visual and language cues in a joint generative model. The proposed method can also provide a textual description for each of the identified semantic steps and video segments. We evaluate this method on a large number of complex YouTube videos and show results of unprecedented quality for this intricate and impactful problem.