Leveraging Auxiliary Information in Text-to-Video Retrieval: A Review

TL;DR

This paper reviews how auxiliary information like visual attributes and contextual data enhances text-to-video retrieval, analyzing 81 studies, their methods, results, and future directions.

Contribution

It provides a comprehensive survey of 81 research papers on auxiliary information in text-to-video retrieval, highlighting methodologies and state-of-the-art results.

Findings

Auxiliary information improves retrieval accuracy.

State-of-the-art methods leverage visual and textual context.

Future research directions include multimodal data integration.

Abstract

Text-to-Video (T2V) retrieval aims to identify the most relevant item from a gallery of videos based on a user's text query. Traditional methods rely solely on aligning video and text modalities to compute the similarity and retrieve relevant items. However, recent advancements emphasise incorporating auxiliary information extracted from video and text modalities to improve retrieval performance and bridge the semantic gap between these modalities. Auxiliary information can include visual attributes, such as objects; temporal and spatial context; and textual descriptions, such as speech and rephrased captions. This survey comprehensively reviews 81 research papers on Text-to-Video retrieval that utilise such auxiliary information. It provides a detailed analysis of their methodologies; highlights state-of-the-art results on benchmark datasets; and discusses available datasets and their auxiliary information. Additionally, it proposes promising directions for future research, focusing on different ways to further enhance retrieval performance using this information.