Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

TL;DR

This paper introduces Contrastive Alignment (CAL), a re-weighting strategy for Vision Language Models that emphasizes visually correlated text tokens, improving cross-modal alignment with minimal computational cost.

Contribution

The paper proposes CAL, a novel contrastive re-weighting method that enhances visual-text alignment by prioritizing tokens with higher visual correlation in VLM training.

Findings

CAL improves VLM performance across multiple benchmarks.

The method is computationally efficient with minimal overhead.

It enhances cross-modal alignment by focusing on visually relevant tokens.

Abstract

Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for assigning distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies. Codes are available at https://github.com/foundation-multimodal-models/CAL.