Diffusion Features to Bridge Domain Gap for Semantic Segmentation

TL;DR

This paper introduces DIFF, a novel diffusion feature fusion method that leverages pre-trained diffusion models to improve cross-domain semantic segmentation, achieving state-of-the-art results by effectively capturing universal features.

Contribution

The paper presents a new framework that utilizes diffusion models' implicit knowledge and sampling techniques to enhance domain generalization in semantic segmentation.

Findings

Outperforms previous methods in domain generalization tasks.

Achieves state-of-the-art benchmark results.

Effectively captures universal features across domains.

Abstract

Pre-trained diffusion models have demonstrated remarkable proficiency in synthesizing images across a wide range of scenarios with customizable prompts, indicating their effective capacity to capture universal features. Motivated by this, our study delves into the utilization of the implicit knowledge embedded within diffusion models to address challenges in cross-domain semantic segmentation. This paper investigates the approach that leverages the sampling and fusion techniques to harness the features of diffusion models efficiently. We propose DIffusion Feature Fusion (DIFF) as a backbone use for extracting and integrating effective semantic representations through the diffusion process. By leveraging the strength of text-to-image generation capability, we introduce a new training framework designed to implicitly learn posterior knowledge from it. Through rigorous evaluation in the contexts of domain generalization semantic segmentation, we establish that our methodology surpasses preceding approaches in mitigating discrepancies across distinct domains and attains the state-of-the-art (SOTA) benchmark.