Semantic Role Aware Correlation Transformer for Text to Video Retrieval

TL;DR

This paper introduces a semantic role aware correlation transformer that explicitly models intra- and inter-modality relationships in text and video, significantly improving text-to-video retrieval accuracy.

Contribution

It proposes a novel transformer architecture that disentangles semantic roles in text and video to better capture their correlations for retrieval tasks.

Findings

Outperforms current state-of-the-art methods on YouCook2 dataset

Achieves higher metrics in all evaluated categories

Surpasses two other SOTA methods in key metrics

Abstract

With the emergence of social media, voluminous video clips are uploaded every day, and retrieving the most relevant visual content with a language query becomes critical. Most approaches aim to learn a joint embedding space for plain textual and visual contents without adequately exploiting their intra-modality structures and inter-modality correlations. This paper proposes a novel transformer that explicitly disentangles the text and video into semantic roles of objects, spatial contexts and temporal contexts with an attention scheme to learn the intra- and inter-role correlations among the three roles to discover discriminative features for matching at different levels. The preliminary results on popular YouCook2 indicate that our approach surpasses a current state-of-the-art method, with a high margin in all metrics. It also overpasses two SOTA methods in terms of two metrics.