DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

TL;DR

DREAM introduces relevance-based augmentation using large foundation models to enhance video-text retrieval by generating more generalized features from enriched data, leading to improved performance on benchmarks.

Contribution

The paper proposes a novel relevance-based augmentation paradigm leveraging large language and visual models to improve video-text retrieval.

Findings

DREAM outperforms existing methods on multiple benchmarks.

Relevance-based augmentation improves data quality and model generalization.

Large foundation models effectively generate relevant data augmentations.

Abstract

Recent progress in video-text retrieval has been driven largely by advancements in model architectures and training strategies. However, the representation learning capabilities of videotext retrieval models remain constrained by lowquality and limited training data annotations. To address this issue, we present a novel ViDeoText Retrieval Paradigm with RElevance-based AugMentation, namely DREAM, which enhances video and text data using large foundation models to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more robust augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). To further enrich video and text information, we propose a relevance-based augmentation method, where LLMs and VGMs generate and integrate new relevant information into the original data. Leveraging this enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of DREAM over existing methods.