Removing Distributional Discrepancies in Captions Improves Image-Text Alignment

TL;DR

This paper presents a method to improve image-text alignment by generating balanced training data with mixed negative captions, enhancing model understanding of visual-language relationships and outperforming existing methods.

Contribution

The authors introduce a novel data augmentation technique that addresses distribution imbalance in training data, leading to improved image-text alignment performance.

Findings

Significantly outperforms current top methods on various datasets.

Effective in ranking images by text alignment quality.

Enhances model understanding of compositional visual-language relationships.

Abstract

In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language models. Our approach focuses on generating high-quality training datasets for the alignment task by producing mixed-type negative captions derived from positive ones. Critically, we address the distribution imbalance between positive and negative captions to ensure that the alignment model does not depend solely on textual information but also considers the associated images for predicting alignment accurately. By creating this enhanced training data, we fine-tune an existing leading visual-language model to boost its capability in understanding alignment. Our model significantly outperforms current top-performing methods across various datasets. We also demonstrate the applicability of our model by ranking the images generated by text-to-image models based on text alignment. Project page: \url{https://yuheng-li.github.io/LLaVA-score/}