Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval

TL;DR

This paper introduces ARL, a novel framework for partially relevant video retrieval that explicitly models and leverages ambiguity in text-video pairs, improving retrieval accuracy by hierarchical and fine-grained learning.

Contribution

The paper proposes Ambiguity-Restrained representation Learning (ARL), incorporating ambiguity detection and multi-level semantic modeling for better PRVR performance.

Findings

ARL effectively detects ambiguous pairs using uncertainty and similarity criteria.

Hierarchical learning improves semantic understanding of ambiguous text-video pairs.

Fine-grained frame-level modeling enhances retrieval accuracy in untrimmed videos.

Abstract

Partially Relevant Video Retrieval~(PRVR) aims to retrieve a video where a specific segment is relevant to a given text query. Typical training processes of PRVR assume a one-to-one relationship where each text query is relevant to only one video. However, we point out the inherent ambiguity between text and video content based on their conceptual scope and propose a framework that incorporates this ambiguity into the model learning process. Specifically, we propose Ambiguity-Restrained representation Learning~(ARL) to address ambiguous text-video pairs. Initially, ARL detects ambiguous pairs based on two criteria: uncertainty and similarity. Uncertainty represents whether instances include commonly shared context across the dataset, while similarity indicates pair-wise semantic overlap. Then, with the detected ambiguous pairs, our ARL hierarchically learns the semantic relationship via multi-positive contrastive learning and dual triplet margin loss. Additionally, we delve into fine-grained relationships within the video instances. Unlike typical training at the text-video level, where pairwise information is provided, we address the inherent ambiguity within frames of the same untrimmed video, which often contains multiple contexts. This allows us to further enhance learning at the text-frame level. Lastly, we propose cross-model ambiguity detection to mitigate the error propagation that occurs when a single model is employed to detect ambiguous pairs for its training. With all components combined, our proposed method demonstrates its effectiveness in PRVR.