Video and Text Matching with Conditioned Embeddings

TL;DR

This paper introduces a conditioned embedding approach for matching videos and text, which dynamically encodes data based on query context, leading to state-of-the-art results across multiple datasets and applications.

Contribution

It proposes a novel conditioned embedding method that encodes video and text jointly, improving matching accuracy and explainability over traditional independent encoding techniques.

Findings

Achieves state-of-the-art results on five datasets.

Demonstrates effective transfer to video-guided machine translation.

Provides a simple, explainable, and extendable matching framework.

Abstract

We present a method for matching a text sentence from a given corpus to a given video clip and vice versa. Traditionally video and text matching is done by learning a shared embedding space and the encoding of one modality is independent of the other. In this work, we encode the dataset data in a way that takes into account the query's relevant information. The power of the method is demonstrated to arise from pooling the interaction data between words and frames. Since the encoding of the video clip depends on the sentence compared to it, the representation needs to be recomputed for each potential match. To this end, we propose an efficient shallow neural network. Its training employs a hierarchical triplet loss that is extendable to paragraph/video matching. The method is simple, provides explainability, and achieves state-of-the-art results for both sentence-clip and video-text by a sizable margin across five different datasets: ActivityNet, DiDeMo, YouCook2, MSR-VTT, and LSMDC. We also show that our conditioned representation can be transferred to video-guided machine translation, where we improved the current results on VATEX. Source code is available at https://github.com/AmeenAli/VideoMatch.