Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation

TL;DR

This paper enhances diffusion-synthetic training for semantic segmentation by introducing techniques that improve mask quality, scalability, and domain transfer, significantly narrowing the gap with real-data training.

Contribution

It proposes three novel techniques—robust training, prompt augmentation, and LoRA-based domain adaptation—to advance diffusion-synthetic semantic segmentation.

Findings

Improved segmentation accuracy on PASCAL VOC, ImageNet-S, Cityscapes.

Effective domain transfer to auto-driving images.

Close performance gap between synthetic and real training data.

Abstract

The advance of generative models for images has inspired various training techniques for image recognition utilizing synthetic images. In semantic segmentation, one promising approach is extracting pseudo-masks from attention maps in text-to-image diffusion models, which enables real-image-and-annotation-free training. However, the pioneering training method using the diffusion-synthetic images and pseudo-masks, i.e., DiffuMask has limitations in terms of mask quality, scalability, and ranges of applicable domains. To overcome these limitations, this work introduces three techniques for diffusion-synthetic semantic segmentation training. First, reliability-aware robust training, originally used in weakly supervised learning, helps segmentation with insufficient synthetic mask quality. %Second, large-scale pretraining of whole segmentation models, not only backbones, on synthetic ImageNet-1k-class images with pixel-labels benefits downstream segmentation tasks. Second, we introduce prompt augmentation, data augmentation to the prompt text set to scale up and diversify training images with a limited text resources. Finally, LoRA-based adaptation of Stable Diffusion enables the transfer to a distant domain, e.g., auto-driving images. Experiments in PASCAL VOC, ImageNet-S, and Cityscapes show that our method effectively closes gap between real and synthetic training in semantic segmentation.