Enhanced Generative Data Augmentation for Semantic Segmentation via Stronger Guidance

TL;DR

This paper presents a novel data augmentation pipeline using a controllable diffusion model with enhanced prompt and visual reference techniques, significantly improving semantic segmentation performance by generating high-quality, structured synthetic images.

Contribution

The work introduces a new augmentation method leveraging Class-Prompt Appending and Visual Prior Blending to produce more accurate synthetic images for semantic segmentation.

Findings

Improved segmentation accuracy on PASCAL VOC datasets.

Effective generation of diverse, high-quality synthetic images.

Enhanced class balance in training datasets.

Abstract

Data augmentation is crucial for pixel-wise annotation tasks like semantic segmentation, where labeling requires significant effort and intensive labor. Traditional methods, involving simple transformations such as rotations and flips, create new images but often lack diversity along key semantic dimensions and fail to alter high-level semantic properties. To address this issue, generative models have emerged as an effective solution for augmenting data by generating synthetic images. Controllable Generative models offer data augmentation methods for semantic segmentation tasks by using prompts and visual references from the original image. However, these models face challenges in generating synthetic images that accurately reflect the content and structure of the original image due to difficulties in creating effective prompts and visual references. In this work, we introduce an effective data augmentation pipeline for semantic segmentation using Controllable Diffusion model. Our proposed method includes efficient prompt generation using Class-Prompt Appending and Visual Prior Blending to enhance attention to labeled classes in real images, allowing the pipeline to generate a precise number of augmented images while preserving the structure of segmentation-labeled classes. In addition, we implement a class balancing algorithm to ensure a balanced training dataset when merging the synthetic and original images. Evaluation on PASCAL VOC datasets, our pipeline demonstrates its effectiveness in generating high-quality synthetic images for semantic segmentation. Our code is available at https://github.com/chequanghuy/Enhanced-Generative-Data-Augmentation-for-Semantic-Segmentation-via-Stronger-Guidance.