MDMMT: Multidomain Multimodal Transformer for Video Retrieval

TL;DR

This paper introduces MDMMT, a multidomain multimodal transformer that achieves state-of-the-art results in text-to-video retrieval on MSRVTT and LSMDC datasets without finetuning, by effectively combining multiple video caption datasets.

Contribution

The paper proposes a novel multidomain multimodal transformer model that improves video retrieval performance and demonstrates the benefits of training on combined datasets for better generalization.

Findings

State-of-the-art results on MSRVTT and LSMDC benchmarks.

Training on multiple datasets enhances test performance.

Significant dataset overlaps identified between MSRVTT and ActivityNet.

Abstract

We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.