Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis

TL;DR

Follow-Your-Instruction is a multimodal LLM-based framework that automatically synthesizes high-quality multi-dimensional data, significantly enhancing generative models' performance and addressing scalability issues in data collection.

Contribution

It introduces a comprehensive MLLM-driven pipeline for automatic multi-dimensional data synthesis, reducing reliance on manual scene construction and improving scalability and quality.

Findings

Synthetic data boosts baseline model performance.

Framework effectively generates 2D, 3D, and 4D data.

Demonstrates scalability and high quality of generated data.

Abstract

With the growing demands of AI-generated content (AIGC), the need for high-quality, diverse, and scalable data has become increasingly crucial. However, collecting large-scale real-world data remains costly and time-consuming, hindering the development of downstream applications. While some works attempt to collect task-specific data via a rendering process, most approaches still rely on manual scene construction, limiting their scalability and accuracy. To address these challenges, we propose Follow-Your-Instruction, a Multimodal Large Language Model (MLLM)-driven framework for automatically synthesizing high-quality 2D, 3D, and 4D data. Our \textbf{Follow-Your-Instruction} first collects assets and their associated descriptions through multimodal inputs using the MLLM-Collector. Then it constructs 3D layouts, and leverages Vision-Language Models (VLMs) for semantic refinement through multi-view scenes with the MLLM-Generator and MLLM-Optimizer, respectively. Finally, it uses MLLM-Planner to generate temporally coherent future frames. We evaluate the quality of the generated data through comprehensive experiments on the 2D, 3D, and 4D generative tasks. The results show that our synthetic data significantly boosts the performance of existing baseline models, demonstrating Follow-Your-Instruction's potential as a scalable and effective data engine for generative intelligence.