FewMMBench: A Benchmark for Multimodal Few-Shot Learning

TL;DR

FewMMBench is a new benchmark designed to evaluate the few-shot learning abilities of multimodal large language models across diverse tasks, revealing insights into their performance with different prompting strategies and model types.

Contribution

Introduces FewMMBench, a comprehensive benchmark for assessing multimodal LLMs in few-shot scenarios, including diverse tasks and prompting methods, with extensive evaluation of 26 models.

Findings

Instruction-tuned models perform well zero-shot but show limited or negative gains with few-shot or CoT prompts.

Retrieval-based demonstrations and larger context sizes provide minimal improvements.

FewMMBench serves as a rigorous tool for diagnosing and improving multimodal few-shot learning.

Abstract

As multimodal large language models (MLLMs) advance in handling interleaved image-text data, assessing their few-shot learning capabilities remains an open challenge. In this paper, we introduce FewMMBench, a comprehensive benchmark designed to evaluate MLLMs under few-shot conditions, with a focus on In-Context Learning (ICL) and Chain-of-Thought (CoT) prompting. Covering a diverse suite of multimodal understanding tasks, from attribute recognition to temporal reasoning, FewMMBench enables systematic analysis across task types, model families, and prompting strategies. We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings. Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning. Retrieval-based demonstrations and increased context size also yield limited gains. These results highlight FewMMBench as a rigorous testbed for diagnosing and advancing few-shot capabilities in multimodal LLMs. The data is available at: https://huggingface.co/datasets/mustafaa/FewMMBench