PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection

TL;DR

PRISM is a training-free, efficient visual instruction data selection method that addresses redundancy in multimodal datasets by modeling intrinsic visual semantics, significantly reducing tuning time and improving performance.

Contribution

PRISM introduces the first training-free framework for visual instruction selection that models intrinsic visual semantics to remove background influence, enhancing efficiency and effectiveness.

Findings

Reduces data selection and tuning time to 30% of traditional methods.

Surpasses full dataset fine-tuning performance on multiple benchmarks.

Achieves 101.7% relative improvement over baseline.

Abstract

Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a \textit{Global Semantic Drift}, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise \textbf{PRISM}, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30\% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7\% relative improvement over the baseline. The code is available for access via \href{https://github.com/bibisbar/PRISM}{this repository}.