Leveraging per Image-Token Consistency for Vision-Language Pre-training

TL;DR

This paper introduces EPIC, a novel vision-language pre-training method that emphasizes image-token consistency, addressing limitations of existing approaches by focusing on salient tokens and their visual relevance, leading to improved downstream task performance.

Contribution

EPIC proposes a new masking and token consistency strategy that enhances vision-language alignment beyond traditional masked language modeling methods.

Findings

EPIC improves performance when combined with state-of-the-art pre-training methods.

EPIC effectively reduces modality bias in vision-language models.

Experimental results show significant gains on downstream tasks.

Abstract

Most existing vision-language pre-training (VLP) approaches adopt cross-modal masked language modeling (CMLM) to learn vision-language associations. However, we find that CMLM is insufficient for this purpose according to our observations: (1) Modality bias: a considerable amount of masked tokens in CMLM can be recovered with only the language information, ignoring the visual inputs. (2) Under-utilization of the unmasked tokens: CMLM primarily focuses on the masked token but it cannot simultaneously leverage other tokens to learn vision-language associations. To handle those limitations, we propose EPIC (lEveraging Per Image-Token Consistency for vision-language pre-training). In EPIC, for each image-sentence pair, we mask tokens that are salient to the image (i.e., Saliency-based Masking Strategy) and replace them with alternatives sampled from a language model (i.e., Inconsistent Token Generation Procedure), and then the model is required to determine for each token in the sentence whether it is consistent with the image (i.e., Image-Token Consistency Task). The proposed EPIC method is easily combined with pre-training methods. Extensive experiments show that the combination of the EPIC method and state-of-the-art pre-training approaches, including ViLT, ALBEF, METER, and X-VLM, leads to significant improvements on downstream tasks. The code is released at https://github.com/gyhdog99/epic.