COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

TL;DR

COSA introduces a novel pretraining approach for vision-language models by concatenating image-text pairs to simulate long-form video data, enhancing temporal understanding and improving performance across multiple downstream tasks.

Contribution

It presents a new method of pretraining that leverages concatenated image-text pairs to model temporal cues without requiring actual video data.

Findings

Improves performance on video-text and image-text tasks.

Achieves state-of-the-art results on several benchmarks.

Enhances temporal modeling in vision-language understanding.

Abstract

Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA jointly models visual contents and event-level temporal cues using only image-text corpora. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo long-form video-paragraph corpus, enabling richer scene transformations and explicit event-description correspondence. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of downstream tasks, including long-form/short-form video-text tasks and image-text tasks such as retrieval, captioning, and question answering. Notably, COSA achieves state-of-the-art results on various competitive benchmarks. Code and model are released at https://github.com/TXH-mercury/COSA.