An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

TL;DR

This paper empirically investigates the effects of scaling, data mixing, and training methods on large multimodal models' performance, demonstrating consistent improvements and providing insights for future research.

Contribution

It presents the first large-scale study of multimodal models up to 70B parameters, evaluating various training techniques and data strategies for visual instruction tuning.

Findings

Scaling models improves performance and language capabilities.

LoRA/QLoRA tuning matches full-model fine-tuning performance.

Higher image resolution and data mixing enhance model effectiveness.

Abstract

Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open-source LMM are performed using models with 13B parameters or smaller. In this paper we present an empirical study of scaling LLaVA up to 33B and 65B/70B, and share our findings from our explorations in image resolution, data mixing and parameter-efficient training methods such as LoRA/QLoRA. These are evaluated by their impact on the multi-modal and language capabilities when completing real-world tasks in the wild. We find that scaling LMM consistently enhances model performance and improves language capabilities, and performance of LoRA/QLoRA tuning of LMM are comparable to the performance of full-model fine-tuning. Additionally, the study highlights the importance of higher image resolutions and mixing multimodal-language data to improve LMM performance, and visual instruction tuning can sometimes improve LMM's pure language capability. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research. Code and checkpoints will be made public.