X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

TL;DR

X$^2$-VLM is a versatile, unified pre-trained model that learns multi-grained vision-language alignments for both images and videos, achieving state-of-the-art results across multiple tasks with high transferability.

Contribution

The paper introduces X$^2$-VLM, a modular, all-in-one pre-training framework that unifies image-text and video-text learning with multi-grained alignments and localization.

Findings

X$^2$-VLM outperforms existing models on image-text and video-text tasks.

The modular design enables high transferability across languages and domains.

Replacing the text encoder with XLM-R enhances multilingual performance.

Abstract

Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others utilize pre-trained object detectors to leverage vision language alignments at the object level. In this paper, we propose to learn multi-grained vision language alignments by a unified pre-training framework that learns multi-grained aligning and multi-grained localization simultaneously. Based on it, we present X$^2$-VLM, an all-in-one model with a flexible modular architecture, in which we further unify image-text pre-training and video-text pre-training in one model. X$^2$-VLM is able to learn unlimited visual concepts associated with diverse text descriptions. Experiment results show that X$^2$-VLM performs the best on base and large scale for both image-text and video-text tasks, making a good trade-off between performance and model scale. Moreover, we show that the modular design of X$^2$-VLM results in high transferability for it to be utilized in any language or domain. For example, by simply replacing the text encoder with XLM-R, X$^2$-VLM outperforms state-of-the-art multilingual multi-modal pre-trained models without any multilingual pre-training. The code and pre-trained models are available at https://github.com/zengyan-97/X2-VLM.