Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativeness

TL;DR

This paper introduces DataTailor, a data selection framework for multi-modal large language models that improves efficiency by selecting only the most informative, unique, and representative data, reducing training costs significantly.

Contribution

DataTailor is a novel data selection method based on informativeness, uniqueness, and representativeness, automatically adapting to datasets without hyperparameter tuning.

Findings

Achieves 101.3% of full-data performance with only 15% of data

Reduces computational costs while maintaining high accuracy

Demonstrates effectiveness across various benchmarks

Abstract

Instruction tuning fine-tunes pre-trained Multi-modal Large Language Models (MLLMs) to handle real-world tasks. However, the rapid expansion of visual instruction datasets introduces data redundancy, leading to excessive computational costs. We propose a collaborative framework, DataTailor, which leverages three key principles--informativeness, uniqueness, and representativeness--for effective data selection. We argue that a valuable sample should be informative of the task, non-redundant, and represent the sample distribution (i.e., not an outlier). We further propose practical ways to score against each principle, which automatically adapts to a given dataset without tedious hyperparameter tuning. Comprehensive experiments on various benchmarks demonstrate that DataTailor achieves 101.3% of the performance of full-data fine-tuning with only 15% of the data, significantly reducing computational costs while maintaining superior results. This exemplifies the "Less is More" philosophy in MLLM development. The code and data is available in this \href{https://github.com/Yuqifan1117/DataTailor}{URL}.