MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

TL;DR

MM-Gen is a scalable data curation method that generates high-quality, task-specific synthetic data to improve vision-language models' performance on specialized tasks like diagram understanding.

Contribution

Introduces MM-Gen, a three-stage targeted synthetic data generation process that significantly enhances VLMs on specialized tasks compared to general datasets.

Findings

29% improvement on spatial reasoning tasks

15% improvement on diagram understanding

Up to 1.6x better than human-curated data

Abstract

Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1.5 struggles with chart and diagram understanding due to scarce task-specific training data. Existing training data, sourced from general-purpose datasets, fails to capture the nuanced details needed for these tasks. We introduce MM-Gen, a scalable method that generates task-specific, high-quality synthetic text for candidate images by leveraging stronger models. MM-Gen employs a three-stage targeted process: partitioning data into subgroups, generating targeted text based on task descriptions, and filtering out redundant and outlier data. Fine-tuning VLMs with data generated by MM-Gen leads to significant performance gains, including 29% on spatial reasoning and 15% on diagram understanding for Llava-1.5 (7B). Compared to human-curated caption data, MM-Gen achieves up to 1.6x better improvements for the original models, proving its effectiveness in enhancing task-specific VLM performance and bridging the gap between general-purpose datasets and specialized requirements. Code available at https://github.com/sjoshi804/MM-Gen.