AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant

TL;DR

AssistSR introduces a new task and dataset for task-oriented video segment retrieval based on multimodal queries, aiming to enhance personal AI assistants in understanding and retrieving instructional video segments.

Contribution

The paper proposes the TQVSR task, creates the AssistSR dataset, and develops the DME model, advancing multimodal video retrieval for personal AI applications.

Findings

DME significantly outperforms baseline methods

AssistSR dataset contains 3.2k questions on 1.6k video segments

Detailed ablation studies validate the model's effectiveness

Abstract

It is still a pipe dream that personal AI assistants on the phone and AR glasses can assist our daily life in addressing our questions like ``how to adjust the date for this watch?'' and ``how to set its heating duration? (while pointing at an oven)''. The queries used in conventional tasks (i.e. Video Question Answering, Video Retrieval, Moment Localization) are often factoid and based on pure text. In contrast, we present a new task called Task-oriented Question-driven Video Segment Retrieval (TQVSR). Each of our questions is an image-box-text query that focuses on affordance of items in our daily life and expects relevant answer segments to be retrieved from a corpus of instructional video-transcript segments. To support the study of this TQVSR task, we construct a new dataset called AssistSR. We design novel guidelines to create high-quality samples. This dataset contains 3.2k multimodal questions on 1.6k video segments from instructional videos on diverse daily-used items. To address TQVSR, we develop a simple yet effective model called Dual Multimodal Encoders (DME) that significantly outperforms several baseline methods while still having large room for improvement in the future. Moreover, we present detailed ablation analyses. Code and data are available at \url{https://github.com/StanLei52/TQVSR}.