PRVR: Partially Relevant Video Retrieval

TL;DR

This paper introduces Partially Relevant Video Retrieval (PRVR), a new task addressing retrieval of videos where only parts are relevant to a query, using a multi-scale similarity learning approach.

Contribution

It formulates PRVR as a multiple instance learning problem and proposes the MS-SL++ network to jointly learn clip- and frame-scale similarities.

Findings

Effective on three diverse datasets

Outperforms existing methods in partial relevance scenarios

Demonstrates viability of multi-scale similarity learning

Abstract

In current text-to-video retrieval (T2VR), videos to be retrieved have been properly trimmed so that a correspondence between the videos and ad-hoc textual queries naturally exists. Note in practice that videos circulated on the Internet and social media platforms, while being relatively short, are typically rich in their content. Often, multiple scenes / actions / events are shown in a single video, leading to a more challenging T2VR setting wherein only part of the video content is relevant w.r.t. a given query. This paper presents a first study on this setting which we term Partially Relevant Video Retrieval (PRVR). Considering that a video typically consists of multiple moments, a video is regarded as partially relevant w.r.t. to a given query if it contains a query-related moment. We formulate the PRVR task as a multiple instance learning problem, and propose a Multi-Scale Similarity Learning (MS-SL++) network that jointly learns both clip-scale and frame-scale similarities to determine the partial relevance between video-query pairs. Extensive experiments on three diverse video-text datasets (TVshow Retrieval, ActivityNet-Captions and Charades-STA) demonstrate the viability of the proposed method.