Free-Mask: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editing

TL;DR

Free-Mask introduces an innovative framework combining diffusion models and image editing to generate diverse, realistic synthetic datasets with accurate segmentation masks, reducing manual annotation efforts and improving segmentation performance.

Contribution

It presents a novel integration of segmentation diffusion models with image editing to generate multi-object, realistic synthetic datasets for improved segmentation training.

Findings

Synthetic data from Free-Mask outperforms real data in segmentation tasks.

Achieves state-of-the-art results on unseen classes in VOC 2012.

Enables zero-shot segmentation performance improvements.

Abstract

Current semantic segmentation models typically require a substantial amount of manually annotated data, a process that is both time-consuming and resource-intensive. Alternatively, leveraging advanced text-to-image models such as Midjourney and Stable Diffusion has emerged as an efficient strategy, enabling the automatic generation of synthetic data in place of manual annotations. However, previous methods have been limited to generating single-instance images, as the generation of multiple instances with Stable Diffusion has proven unstable. To address this limitation and expand the scope and diversity of synthetic datasets, we propose a framework \textbf{Free-Mask} that combines a Diffusion Model for segmentation with advanced image editing capabilities, allowing for the integration of multiple objects into images via text-to-image models. Our method facilitates the creation of highly realistic datasets that closely emulate open-world environments while generating accurate segmentation masks. It reduces the labor associated with manual annotation and also ensures precise mask generation. Experimental results demonstrate that synthetic data generated by \textbf{Free-Mask} enables segmentation models to outperform those trained on real data, especially in zero-shot settings. Notably, \textbf{Free-Mask} achieves new state-of-the-art results on previously unseen classes in the VOC 2012 benchmark.