Rethinking Overlooked Aspects in Vision-Language Models

TL;DR

This paper investigates the impact of data efficiency in pre-training and instruction tuning for vision-language models, revealing that more data isn't always better and identifying optimal datasets for improved performance.

Contribution

It introduces a pipeline to identify the most effective instruction tuning datasets and emphasizes optimizing data usage rather than solely increasing data size.

Findings

Increasing pre-training data size can degrade performance.

Not all instruction tuning datasets are necessary for optimal results.

A pipeline to select efficient instruction tuning datasets is proposed.

Abstract

Recent advancements in large vision-language models (LVLMs), such as GPT4-V and LLaVA, have been substantial. LLaVA's modular architecture, in particular, offers a blend of simplicity and efficiency. Recent works mainly focus on introducing more pre-training and instruction tuning data to improve model's performance. This paper delves into the often-neglected aspects of data efficiency during pre-training and the selection process for instruction tuning datasets. Our research indicates that merely increasing the size of pre-training data does not guarantee improved performance and may, in fact, lead to its degradation. Furthermore, we have established a pipeline to pinpoint the most efficient instruction tuning (SFT) dataset, implying that not all SFT data utilized in existing studies are necessary. The primary objective of this paper is not to introduce a state-of-the-art model, but rather to serve as a roadmap for future research, aiming to optimize data usage during pre-training and fine-tuning processes to enhance the performance of vision-language models.