SynthSeg-Agents: Multi-Agent Synthetic Data Generation for Zero-Shot Weakly Supervised Semantic Segmentation

TL;DR

This paper introduces SynthSeg Agents, a multi-agent framework driven by Large Language Models to generate synthetic training data for zero-shot weakly supervised semantic segmentation, eliminating the need for real images.

Contribution

It proposes a novel multi-agent system utilizing LLMs and vision-language models to generate high-quality synthetic data for semantic segmentation without real image supervision.

Findings

Achieves competitive segmentation performance on PASCAL VOC 2012 and COCO 2014 datasets.

Demonstrates the effectiveness of LLM-driven synthetic data generation in zero-shot settings.

Reduces reliance on real annotated images for training semantic segmentation models.

Abstract

Weakly Supervised Semantic Segmentation (WSSS) with image level labels aims to produce pixel level predictions without requiring dense annotations. While recent approaches have leveraged generative models to augment existing data, they remain dependent on real world training samples. In this paper, we introduce a novel direction, Zero Shot Weakly Supervised Semantic Segmentation (ZSWSSS), and propose SynthSeg Agents, a multi agent framework driven by Large Language Models (LLMs) to generate synthetic training data entirely without real images. SynthSeg Agents comprises two key modules, a Self Refine Prompt Agent and an Image Generation Agent. The Self Refine Prompt Agent autonomously crafts diverse and semantically rich image prompts via iterative refinement, memory mechanisms, and prompt space exploration, guided by CLIP based similarity and nearest neighbor diversity filtering. These prompts are then passed to the Image Generation Agent, which leverages Vision Language Models (VLMs) to synthesize candidate images. A frozen CLIP scoring model is employed to select high quality samples, and a ViT based classifier is further trained to relabel the entire synthetic dataset with improved semantic precision. Our framework produces high quality training data without any real image supervision. Experiments on PASCAL VOC 2012 and COCO 2014 show that SynthSeg Agents achieves competitive performance without using real training images. This highlights the potential of LLM driven agents in enabling cost efficient and scalable semantic segmentation.