Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?

TL;DR

This paper introduces CLIPPER, a novel data selection method for multimodal LLM fine-tuning that uses in-context learning to efficiently identify representative datasets, reducing costs while maintaining performance.

Contribution

CLIPPER is a training-free, in-context learning-based data selection pipeline that improves data efficiency and diversity in task-specific fine-tuning of multimodal LLMs.

Findings

CLIPPER achieves 47% data efficiency on VRSBench.

Llama-3.2-11B-Vision-Instruct training time is reduced by 37%.

Matches full fine-tuning performance with lower computational costs.

Abstract

Injecting world knowledge into pretrained multimodal large language models (MLLMs) is essential for domain-specific applications. Task-specific fine-tuning achieves this by tailoring MLLMs to high-quality in-domain data but encounters scalability challenges as datasets grow, necessitating a trade-off between performance and computational overhead. Existing data selection methods rely on additional scoring models or heuristic clustering, failing to concentrate on both data importance and diversity. Moreover, both methods overlook the interplay among training samples. To address these limitations, we propose CLIPPER, a training-free data selection pipeline that separates parameter and world knowledge, and leverages in-context learning to probe model responses to different demonstration-query combinations. CLIPPER identifies coresets that mirror the original dataset's perplexity distribution, preserving critical samples while maintaining diversity. Experiments on two MLLMs and three datasets show that CLIPPER matches full fine-tuning performance with significantly lower costs: Qwen2.5-VL-7B attains 47% data efficiency on VRSBench, and Llama-3.2-11B-Vision-Instruct reduces ScienceQA training time by 37%.