Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark

TL;DR

This paper introduces Repurpose-10K, a large-scale dataset and benchmark for video repurposing from user-generated content, along with a baseline model that integrates multi-modal data for short-form video creation.

Contribution

The paper presents a novel large-scale dataset and benchmark for video repurposing, along with a two-stage annotation process and a cross-modal fusion baseline model.

Findings

Repurpose-10K contains over 10,000 videos and 120,000 annotated clips.

The baseline model effectively integrates audio, visual, and caption data.

The dataset and benchmark facilitate research in video repurposing from user-generated content.

Abstract

The demand for producing short-form videos for sharing on social media platforms has experienced significant growth in recent times. Despite notable advancements in the fields of video summarization and highlight detection, which can create partially usable short films from raw videos, these approaches are often domain-specific and require an in-depth understanding of real-world video content. To tackle this predicament, we propose Repurpose-10K, an extensive dataset comprising over 10,000 videos with more than 120,000 annotated clips aimed at resolving the video long-to-short task. Recognizing the inherent constraints posed by untrained human annotators, which can result in inaccurate annotations for repurposed videos, we propose a two-stage solution to obtain annotations from real-world user-generated content. Furthermore, we offer a baseline model to address this challenging task by integrating audio, visual, and caption aspects through a cross-modal fusion and alignment framework. We aspire for our work to ignite groundbreaking research in the lesser-explored realms of video repurposing.