WeatherDG: LLM-assisted Diffusion Model for Procedural Weather Generation in Domain-Generalized Semantic Segmentation

TL;DR

WeatherDG leverages LLMs and diffusion models to generate diverse, weather-specific images for driving scenarios, enhancing domain generalization in semantic segmentation tasks.

Contribution

The paper introduces WeatherDG, a novel framework combining LLMs and diffusion models for weather-diverse image generation to improve segmentation model generalization.

Findings

Significant improvement in segmentation performance on multiple datasets.

13.9% mIoU gain in Cityscapes to ACDC transfer.

Enhanced object generation under various weather conditions.

Abstract

In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e, Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU.