Less is More: High-value Data Selection for Visual Instruction Tuning

TL;DR

This paper introduces TIVE, a data selection method that reduces visual instruction data by 85% based on influence and difficulty scores, maintaining or improving model performance across multiple benchmarks.

Contribution

The paper presents a novel high-value data selection approach for visual instruction tuning that significantly reduces training data and cost while preserving or enhancing model performance.

Findings

Using only 15% of data achieves comparable performance to full datasets.

Redundant data within visual instruction datasets can be effectively eliminated.

TIVE outperforms full-data models on several benchmarks.

Abstract

Visual instruction tuning is the key to building large vision language models~(LVLMs), which can greatly improve the task generalization and solving capabilities by learning a mixture of instruction data from diverse visual tasks. Previous work mostly collects multiple existing visual instruction datasets via heuristic ways for training (even more than a million instructions), which may introduce data redundancy and enlarge the training cost. To investigate this issue, we conduct a series of empirical studies, which reveal a significant redundancy within the visual instruction datasets, and show that greatly reducing the amount of instructions from several tasks even do not affect the performance. Based on the findings, we propose a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. In TIVE, we first estimate the instance influence score on its corresponding task, and the task difficulty score, based on the gradient-based influence functions. Then, we leverage the two kinds of scores to determine the task proportion within the selected visual instruction subset, and select high-value instances for each task, respectively. Experiments on various LVLMs show that our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks, even surpassing it on four of the benchmarks. Our code and data will be publicly released.