Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training

TL;DR

This paper introduces Generate Any Scene, a data engine that creates diverse, complex scene graphs for training visual generation models, improving their compositional understanding and semantic alignment through synthetic data and iterative self-improvement.

Contribution

The paper presents a novel scene graph-based data synthesis framework that enhances visual generation models' performance and semantic accuracy, including a self-improving training loop, a distillation method, and a low-cost reward model.

Findings

Stable Diffusion v1.5 improves by 4% with generated data.

Fewer than 800 synthetic captions boost compositional generation by 10%.

The reward model surpasses CLIP-based methods by 5%.

Abstract

Recent advances in text-to-vision generation excel in visual fidelity but struggle with compositional generalization and semantic alignment. Existing datasets are noisy and weakly compositional, limiting models' understanding of complex scenes, while scalable solutions for dense, high-quality annotations remain a challenge. We introduce Generate Any Scene, a data engine that systematically enumerates scene graphs representing the combinatorial array of possible visual scenes. Generate Any Scene dynamically constructs scene graphs of varying complexity from a structured taxonomy of objects, attributes, and relations. Given a sampled scene graph, Generate Any Scene translates it into a caption for text-to-image or text-to-video generation; it also translates it into a set of visual question answers that allow automatic evaluation and reward modeling of semantic alignment. Using Generate Any Scene, we first design a self-improving framework where models iteratively enhance their performance using generated data. Stable Diffusion v1.5 achieves an average 4% improvement over baselines and surpassing fine-tuning on CC3M. Second, we also design a distillation algorithm to transfer specific strengths from proprietary models to their open-source counterparts. Using fewer than 800 synthetic captions, we fine-tune Stable Diffusion v1.5 and achieve a 10% increase in TIFA score on compositional and hard concept generation. Third, we create a reward model to align model generation with semantic accuracy at a low cost. Using GRPO algorithm, we fine-tune SimpleAR-0.5B-SFT and surpass CLIP-based methods by +5% on DPG-Bench. Finally, we apply these ideas to the downstream task of content moderation where we train models to identify challenging cases by learning from synthetic data.