Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

TL;DR

This paper systematically analyzes multimodal in-context learning, revealing its limitations in transfer and alignment, and proposes an inference-stage enhancement to improve task mapping transfer.

Contribution

It provides a detailed decomposition of multimodal ICL, identifies key bottlenecks, and introduces a simple method to enhance task mapping transfer during inference.

Findings

Multimodal ICL performs comparably to text-only ICL in zero-shot but degrades in few-shot settings.

Models lack reasoning-level alignment between visual and textual representations.

A simple inference-stage enhancement improves task mapping transfer.

Abstract

In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available \href{https://github.com/deeplearning-wisc/Multimocal-ICL-Analysis-Framework-MGI}{here}.