Towards Robust Instruction Tuning on Multimodal Large Language Models

TL;DR

This paper introduces INSTRAUG, an automatic instruction augmentation method for multimodal large language models, significantly expanding instruction datasets and improving model alignment across multiple tasks without extensive human effort.

Contribution

The paper presents INSTRAUG, a novel automatic data augmentation technique that expands instruction datasets by 30 times for multimodal tasks, enhancing model performance.

Findings

INSTRAUG increases dataset size by 30 times.

Improves model alignment across 12 multimodal tasks.

Achieves benefits comparable to large-scale data scaling.

Abstract

Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.