ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation

TL;DR

ViLTA introduces a novel approach to vision-language pre-training that enhances model robustness and convergence speed through cross-distillation for MLM and hard negative synthesis for ITM, leading to improved performance.

Contribution

The paper proposes ViLTA, a new method with two components that improve fine-grained representation learning and robustness in vision-language pre-training.

Findings

Achieves better performance on vision-language benchmarks.

Enhances model robustness via cross-distillation in MLM.

Speeds up convergence with hard negative mining in ITM.

Abstract

Vision-language pre-training (VLP) methods are blossoming recently, and its crucial goal is to jointly learn visual and textual features via a transformer-based architecture, demonstrating promising improvements on a variety of vision-language tasks. Prior arts usually focus on how to align visual and textual features, but strategies for improving the robustness of model and speeding up model convergence are left insufficiently explored. In this paper, we propose a novel method ViLTA, comprising of two components to further facilitate the model to learn fine-grained representations among image-text pairs. For Masked Language Modeling (MLM), we propose a cross-distillation method to generate soft labels to enhance the robustness of model, which alleviates the problem of treating synonyms of masked words as negative samples in one-hot labels. For Image-Text Matching (ITM), we leverage the current language encoder to synthesize hard negatives based on the context of language input, encouraging the model to learn high-quality representations by increasing the difficulty of the ITM task. By leveraging the above techniques, our ViLTA can achieve better performance on various vision-language tasks. Extensive experiments on benchmark datasets demonstrate that the effectiveness of ViLTA and its promising potential for vision-language pre-training.