Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment

TL;DR

This paper introduces MIVPG, a novel method that enhances multimodal large language models by leveraging instance correlation in visual data, leading to improved performance across various vision-language tasks.

Contribution

The paper proposes MIVPG, a general component that enriches visual representations in LLMs by exploiting instance correlation, addressing limitations of existing query-based transformers like Q-former.

Findings

MIVPG improves performance on multiple vision-language datasets.

Enriched visual representations lead to better task accuracy.

The method effectively captures instance correlation for enhanced visual understanding.

Abstract

Multimodal Large Language Models (MLLMs) have achieved SOTA performance in various visual language tasks by fusing the visual representations with LLMs leveraging some visual adapters. In this paper, we first establish that adapters using query-based Transformers such as Q-former is a simplified Multi-instance Learning method without considering instance heterogeneity/correlation. We then propose a general component termed Multi-instance Visual Prompt Generator (MIVPG) to incorporate enriched visual representations into LLMs by taking advantage of instance correlation between images or patches for the same sample. Quantatitive evaluation on three public vision-language (VL) datasets from different scenarios shows that the proposed MIVPG improves Q-former in main VL tasks.