MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

TL;DR

This paper introduces MMInstruct, a large, diverse, high-quality visual instruction dataset created using semi-automatic generation with GPT models, significantly improving vision-language model performance across multiple benchmarks.

Contribution

The paper presents MMInstruct, a novel multi-domain visual instruction dataset generated with an efficient, semi-automatic method leveraging GPT models, enhancing model performance and diversity.

Findings

Model fine-tuning on MMInstruct achieves state-of-the-art results on 10 out of 12 benchmarks.

The instruction generation process is low-cost and scalable, reducing manual effort.

MMInstruct significantly improves the diversity and quality of visual instruction data.

Abstract

Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct.