Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining

TL;DR

This paper introduces a novel framework that generates challenging negative samples in both image and text modalities to improve the compositional reasoning abilities of visual language models, addressing limitations of traditional contrastive training.

Contribution

It proposes a dual-direction negative sample mining and generation approach, significantly enhancing VLMs' performance on complex multimodal reasoning tasks.

Findings

Improved accuracy on compositional reasoning benchmarks

Enhanced discrimination of complex image-text interactions

Effective negative sample generation in both modalities

Abstract

Contemporary large-scale visual language models (VLMs) exhibit strong representation capacities, making them ubiquitous for enhancing image and text understanding tasks. They are often trained in a contrastive manner on a large and diverse corpus of images and corresponding text captions scraped from the internet. Despite this, VLMs often struggle with compositional reasoning tasks which require a fine-grained understanding of the complex interactions of objects and their attributes. This failure can be attributed to two main factors: 1) Contrastive approaches have traditionally focused on mining negative examples from existing datasets. However, the mined negative examples might not be difficult for the model to discriminate from the positive. An alternative to mining would be negative sample generation 2) But existing generative approaches primarily focus on generating hard negative texts associated with a given image. Mining in the other direction, i.e., generating negative image samples associated with a given text has been ignored. To overcome both these limitations, we propose a framework that not only mines in both directions but also generates challenging negative samples in both modalities, i.e., images and texts. Leveraging these generative hard negative samples, we significantly enhance VLMs' performance in tasks involving multimodal compositional reasoning. Our code and dataset are released at https://ugorsahin.github.io/enhancing-multimodal-compositional-reasoning-of-vlm.html.