How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?

TL;DR

This paper investigates how textual information influences the selection of in-context examples in multimodal large language models, revealing modality sensitivities and proposing a supervised retrieval method to improve in-context learning performance.

Contribution

It introduces MSIER, a neural network-based supervised retriever that enhances in-context learning in MLLMs by effectively leveraging textual and visual data.

Findings

Retriever performance is highly sensitive to modality choices.

MSIER outperforms unsupervised methods across multiple tasks.

Modality factors significantly affect training and effectiveness of retrieval models.

Abstract

The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.