Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

TL;DR

Infinity-MM introduces a large-scale, high-quality multimodal instruction dataset and a synthetic data generation method, enabling training of a 2-billion-parameter VLM that achieves state-of-the-art results in multimodal tasks.

Contribution

The paper presents a new extensive multimodal instruction dataset and a synthetic data generation approach, significantly improving VLM performance and scalability.

Findings

Achieved SOTA performance with Aquila-VL-2B on multimodal tasks.

Created over 40 million high-quality instruction samples.

Demonstrated effective large-scale instruction data synthesis.

Abstract

Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM.