Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?

TL;DR

This paper investigates whether multimodal large language models truly perform multimodal in-context learning or if their success is mainly driven by textual content, proposing a new demo selection method to enhance performance.

Contribution

The study reveals that multimodal ICL is primarily influenced by textual information and introduces MMICES, a demo selection strategy considering both visual and textual modalities.

Findings

Multimodal ICL is mainly driven by textual content.

Visual content aids in selecting better demos.

Proposed MMICES improves ICL performance.

Abstract

Large Language Models (LLMs) with in-context learning (ICL) ability can quickly adapt to a specific context given a few demonstrations (demos). Recently, Multimodal Large Language Models (MLLMs) built upon LLMs have also shown multimodal ICL ability, i.e., responding to queries given a few multimodal demos, including images, queries, and answers. While ICL has been extensively studied on LLMs, its research on MLLMs remains limited. One essential question is whether these MLLMs can truly conduct multimodal ICL, or if only the textual modality is necessary. We investigate this question by examining two primary factors that influence ICL: 1) Demo content, i.e., understanding the influences of demo content in different modalities. 2) Demo selection strategy, i.e., how to select better multimodal demos for improved performance. Experiments revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence. Interestingly, visual content is still necessary and useful for selecting demos to increase performance. Motivated by our analysis, we propose a simple yet effective approach, termed Mixed Modality In-Context Example Selection (MMICES), which considers both visual and language modalities when selecting demos. Extensive experiments are conducted to support our findings and verify the improvement brought by our method. Code is available at \url{https://chenxshuo.github.io/m-icl/}.