From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning

TL;DR

This paper systematically analyzes how and why multimodal in-context learning works in large language models, revealing the importance of modality impact and proposing strategies to improve performance across diverse tasks.

Contribution

It provides a principled evaluation of multimodal ICL, uncovering modality-specific effects, and offers demonstration strategies to enhance model performance based on task-specific modality importance.

Findings

Modalities impact differently across tasks in multimodal ICL.

Modality-driven demonstration strategies improve ICL performance.

Models exhibit inductive biases influenced by multimodal ICL, even contradicting pretraining priors.

Abstract

Motivated by in-context learning (ICL) capabilities of Large Language Models (LLMs), multimodal LLMs with additional visual modality are also exhibited with similar ICL abilities when multiple image-text pairs are provided as demonstrations. However, relatively less work has been done to investigate the principles behind how and why multimodal ICL works. We conduct a systematic and principled evaluation of multimodal ICL for models of different scales on a broad spectrum of new yet critical tasks. Through perturbations over different modality information, we show that modalities matter differently across tasks in multimodal ICL. Guided by task-specific modality impact, we recommend modality-driven demonstration strategies to boost ICL performance. We also find that models may follow inductive biases from multimodal ICL even if they are rarely seen in or contradict semantic priors from pretraining data. Our principled analysis provides a comprehensive way of understanding the role of demonstrations in multimodal in-context learning, and sheds light on effectively improving multimodal ICL on a wide range of tasks.