FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models

TL;DR

This paper identifies and addresses key issues in contrastive vision-language pre-training, such as negative pair assignment and caption quality, leading to significant performance improvements across multiple datasets.

Contribution

The authors propose solutions involving multiple true positive pairs and sigmoid loss, substantially enhancing vision-language model performance.

Findings

+6% average improvement in image recognition across 11 datasets

+19% improvement in Flickr30k image retrieval

+15% improvement in MSCOCO image retrieval

Abstract

Despite noise and caption quality having been acknowledged as important factors impacting vision-language contrastive pre-training, in this paper, we show that the full potential of improving the training process by addressing such issues is yet to be realized. Specifically, we firstly study and analyze two issues affecting training: incorrect assignment of negative pairs, and low caption quality and diversity. Then, we devise effective solutions for addressing both problems, which essentially require training with multiple true positive pairs. Finally, we propose training with sigmoid loss to address such a requirement. We show very large gains over the current state-of-the-art for both image recognition ($\sim +6\%$ on average over 11 datasets) and image retrieval ($\sim +19\%$ on Flickr30k and $\sim +15\%$ on MSCOCO).