BlendFusion -- Scalable Synthetic Data Generation for Diffusion Model Training

TL;DR

BlendFusion is a scalable framework that generates high-quality synthetic image-caption data from 3D scenes, addressing issues of visual inconsistency and model collapse in diffusion model training.

Contribution

It introduces a novel pipeline with object-centric camera placement, filtering, and captioning, and provides the curated FineBLEND dataset for diffusion model training.

Findings

FineBLEND dataset is diverse and high-quality.

Object-centric camera placement improves data quality.

The framework is highly configurable for community use.

Abstract

With the rapid adoption of diffusion models, synthetic data generation has emerged as a promising approach for addressing the growing demand for large-scale image datasets. However, images generated purely by diffusion models often exhibit visual inconsistencies, and training models on such data can create an autophagous feedback loop that leads to model collapse, commonly referred to as Model Autophagy Disorder (MAD). To address these challenges, we propose BlendFusion, a scalable framework for synthetic data generation from 3D scenes using path tracing. Our pipeline incorporates an object-centric camera placement strategy, robust filtering mechanisms, and automatic captioning to produce high-quality image-caption pairs. Using this pipeline, we curate FineBLEND, an image-caption dataset constructed from a diverse set of 3D scenes. We empirically analyze the quality of FineBLEND and compare it to several widely used image-caption datasets. We also demonstrate the effectiveness of our object-centric camera placement strategy relative to object-agnostic sampling approaches. Our open-source framework is designed for high configurability, enabling the community to create their own datasets from 3D scenes.