All in One: A Unified Synthetic Data Pipeline for Multimodal Video Understanding

TL;DR

This paper introduces a unified synthetic data pipeline for multimodal video understanding, enabling scalable data generation for multiple tasks and improving model reasoning abilities, which outperforms traditional methods on real datasets.

Contribution

The authors develop a versatile synthetic data generation framework supporting multiple tasks and introduce a VQA-based fine-tuning strategy to enhance reasoning in multimodal video models.

Findings

Models trained on synthetic data generalize well to real-world datasets.

Synthetic data training often outperforms traditional real-data training.

Unified pipeline supports diverse multimodal video tasks effectively.

Abstract

Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as object counting, question answering, and segmentation. However, collecting and annotating multimodal video data in real-world is costly, slow, and inherently limited in diversity and coverage. To address this challenge, we propose a unified synthetic data generation pipeline capable of automatically producing unlimited multimodal video data with rich and diverse supervision. Our framework supports multiple task formats within a single pipeline, enabling scalable and consistent data creation across tasks. To further enhance reasoning ability, we introduce a VQA-based fine-tuning strategy that trains models to answer structured questions about visual content rather than relying solely on captions or simple instructions. This formulation encourages deeper visual grounding and reasoning. We evaluate our approach in three challenging tasks: video object counting, video-based visual question answering, and video object segmentation. Experimental results demonstrate that models trained predominantly on synthetic data generalize effectively to real-world datasets, often outperforming traditionally trained counterparts. Our findings highlight the potential of unified synthetic data pipelines as a scalable alternative to expensive real-world annotation for multimodal video understanding.