MS4UI: A Dataset for Multi-modal Summarization of User Interface Instructional Videos

TL;DR

This paper introduces MS4UI, a new dataset of 2,413 UI instructional videos with annotations for multi-modal summarization, highlighting the challenges faced by current methods and the need for specialized approaches.

Contribution

The paper presents a novel benchmark dataset for UI instructional video summarization, filling a gap in existing datasets and enabling comprehensive evaluation of multi-modal summarization methods.

Findings

State-of-the-art methods perform poorly on UI video summarization

The dataset enables evaluation of concise and executable video summaries

Highlights the need for new methods tailored to UI instructional videos

Abstract

We study multi-modal summarization for instructional videos, whose goal is to provide users an efficient way to learn skills in the form of text instructions and key video frames. We observe that existing benchmarks focus on generic semantic-level video summarization, and are not suitable for providing step-by-step executable instructions and illustrations, both of which are crucial for instructional videos. We propose a novel benchmark for user interface (UI) instructional video summarization to fill the gap. We collect a dataset of 2,413 UI instructional videos, which spans over 167 hours. These videos are manually annotated for video segmentation, text summarization, and video summarization, which enable the comprehensive evaluations for concise and executable video summarization. We conduct extensive experiments on our collected MS4UI dataset, which suggest that state-of-the-art multi-modal summarization methods struggle on UI video summarization, and highlight the importance of new methods for UI instructional video summarization.