Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

TL;DR

This paper introduces CLIPF, a novel text masking strategy for vision-language model pre-training that leverages word frequency, improving training efficiency and performance especially with fewer input tokens.

Contribution

The paper reveals the importance of word frequency in text masking strategies and proposes CLIPF, a new method that outperforms existing approaches like syntax masking in VLM training.

Findings

CLIPF outperforms syntax masking in various experiments.

Word frequency impacts the effectiveness of text masking strategies.

All strategies perform better with sufficient training epochs.

Abstract

Vision Language Models (VLMs) can be trained more efficiently if training sets can be reduced in size. Recent work has shown the benefits of masking text during VLM training using a variety of strategies (truncation, random masking, block masking and syntax masking) and has reported syntax masking as the top performer. In this paper, we analyze the impact of different text masking strategies on the word frequency in the training data, and show that this impact is connected to model success. This finding motivates Contrastive Language-Image Pre-training with Word Frequency Masking (CLIPF), our proposed masking approach, which directly leverages word frequency. Extensive experiments demonstrate the advantages of CLIPF over syntax masking and other existing approaches, particularly when the number of input tokens decreases. We show that not only CLIPF, but also other existing masking strategies, outperform syntax masking when enough epochs are used during training, a finding of practical importance for selecting a text masking method for VLM training. Our code is available online.