Improved Baselines with Visual Instruction Tuning

TL;DR

This paper demonstrates that simple modifications to LLaVA, including a powerful vision-language connector and targeted data, significantly improve multimodal model performance across multiple benchmarks with minimal data and training time.

Contribution

The authors introduce effective modifications to LLaVA, establishing stronger, data-efficient baselines that achieve state-of-the-art results in visual instruction tuning.

Findings

Achieved state-of-the-art across 11 benchmarks.

Used only 1.2M publicly available data.

Completed training in approximately 1 day on a single 8-A100 node.

Abstract

Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.