TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment

TL;DR

TIPSv2 introduces enhanced patch-text alignment techniques in vision-language models, significantly improving dense correspondence and downstream task performance through novel training strategies and model upgrades.

Contribution

The paper proposes novel methods, including patch-level distillation and an upgraded pretraining objective, to substantially improve dense patch-text alignment in vision-language models.

Findings

Patch-level distillation surpasses teacher models in patch-text alignment.

iBOT++ with unmasked token loss enhances pretraining effectiveness.

TIPSv2 achieves state-of-the-art results on multiple vision-language tasks.

Abstract

Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation and depth prediction. However, a fundamental capability that these models still struggle with is aligning dense patch representations with text embeddings of corresponding concepts. In this work, we investigate this critical issue and propose novel techniques to enhance this capability in foundational vision-language models. First, we reveal that a patch-level distillation procedure significantly boosts dense patch-text alignment -- surprisingly, the patch-text alignment of the distilled student model strongly surpasses that of the teacher model. This observation inspires us to consider modifications to pretraining recipes, leading us to propose iBOT++, an upgrade to the commonly-used iBOT masked image objective, where unmasked tokens also contribute directly to the loss. This dramatically enhances patch-text alignment of pretrained models. Additionally, to improve vision-language pretraining efficiency and effectiveness, we modify the exponential moving average setup in the learning recipe, and introduce a caption sampling strategy to benefit from synthetic captions at different granularities. Combining these components, we develop TIPSv2, a new family of image-text encoder models suitable for a wide range of downstream applications. Through comprehensive experiments on 9 tasks and 20 datasets, we demonstrate strong performance, generally on par with or better than recent vision encoder models. Code and models are released via our project page at https://gdm-tipsv2.github.io/ .