Uniform Masking Prevails in Vision-Language Pretraining

TL;DR

This paper demonstrates that increasing the masking rate in vision-language pretraining enhances downstream task performance and makes simple uniform masking as effective as complex strategies, revealing broader roles of MLM.

Contribution

The study shows that higher masking rates improve performance and reduce differences among masking strategies, challenging the focus on complex masking methods.

Findings

Higher masking rates boost downstream task performance.

Uniform masking becomes competitive with complex strategies.

Increased masking rate benefits Image-Text Matching tasks.

Abstract

Masked Language Modeling (MLM) has proven to be an essential component of Vision-Language (VL) pretraining. To implement MLM, the researcher must make two design choices: the masking strategy, which determines which tokens to mask, and the masking rate, which determines how many tokens to mask. Previous work has focused primarily on the masking strategy while setting the masking rate at a default of 15\%. In this paper, we show that increasing this masking rate improves downstream performance while simultaneously reducing performance gap among different masking strategies, rendering the uniform masking strategy competitive to other more complex ones. Surprisingly, we also discover that increasing the masking rate leads to gains in Image-Text Matching (ITM) tasks, suggesting that the role of MLM goes beyond language modeling in VL pretraining.