ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Models

TL;DR

This paper systematically evaluates how well video retrieval models understand compositional and syntactic components of text queries, revealing that object attributes are more influential than actions and syntax, and that image-text pre-trained models excel in understanding.

Contribution

It provides a comprehensive analysis of the impact of objects, attributes, actions, and syntax on video retrieval performance, highlighting the superiority of image-text pre-trained models in this domain.

Findings

Objects & attributes are more critical than actions and syntax.

Pre-trained image-text models outperform video-text models in understanding.

Syntax and actions have a minor effect on retrieval accuracy.

Abstract

Video retrieval (VR) involves retrieving the ground truth video from the video database given a text caption or vice-versa. The two important components of compositionality: objects & attributes and actions are joined using correct syntax to form a proper text query. These components (objects & attributes, actions and syntax) each play an important role to help distinguish among videos and retrieve the correct ground truth video. However, it is unclear what is the effect of these components on the video retrieval performance. We therefore, conduct a systematic study to evaluate the compositional and syntactic understanding of video retrieval models on standard benchmarks such as MSRVTT, MSVD and DIDEMO. The study is performed on two categories of video retrieval models: (i) which are pre-trained on video-text pairs and fine-tuned on downstream video retrieval datasets (Eg. Frozen-in-Time, Violet, MCQ etc.) (ii) which adapt pre-trained image-text representations like CLIP for video retrieval (Eg. CLIP4Clip, XCLIP, CLIP2Video etc.). Our experiments reveal that actions and syntax play a minor role compared to objects & attributes in video understanding. Moreover, video retrieval models that use pre-trained image-text representations (CLIP) have better syntactic and compositional understanding as compared to models pre-trained on video-text data. The code is available at https://github.com/IntelLabs/multimodal_cognitive_ai/tree/main/ICSVR