LLaVA-Video: Video Instruction Tuning With Synthetic Data

TL;DR

LLaVA-Video is a new video large multimodal model trained on a synthetic dataset, enabling effective video instruction-following without extensive real-world data collection.

Contribution

We created a high-quality synthetic dataset for video instruction tuning and trained a novel video LMM, demonstrating strong benchmark performance.

Findings

Effective video instruction-following achieved

Synthetic data enhances model performance

Dataset and pipeline will be publicly released

Abstract

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.