Visual Semantic Description Generation with MLLMs for Image-Text Matching

TL;DR

This paper introduces a novel framework using multimodal large language models to generate visual semantic descriptions, significantly improving image-text matching performance and zero-shot generalization across domains.

Contribution

It proposes a new method that leverages MLLMs for semantic parsing to enhance cross-modal alignment in image-text matching tasks.

Findings

Significant performance improvements on Flickr30K and MSCOCO datasets.

Effective zero-shot generalization to cross-domain image-text matching tasks.

Seamless integration with existing ITM models enhances their capabilities.

Abstract

Image-text matching (ITM) aims to address the fundamental challenge of aligning visual and textual modalities, which inherently differ in their representations, continuous, high-dimensional image features vs. discrete, structured text. We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantic parsers. By generating rich Visual Semantic Descriptions (VSD), MLLMs provide semantic anchor that facilitate cross-modal alignment. Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency. These modules can be seamlessly integrated into existing ITM models. Extensive experiments on Flickr30K and MSCOCO demonstrate substantial performance improvements. The approach also exhibits remarkable zero-shot generalization to cross-domain tasks, including news and remote sensing ITM. The code and model checkpoints are available at https://github.com/Image-Text-Matching/VSD.