G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models

TL;DR

This paper introduces G4Seg, a novel segmentation refinement method leveraging diffusion model generative priors to improve inexact segmentation by analyzing pattern discrepancies between original and generated images.

Contribution

G4Seg is the first to utilize diffusion model generative priors for inexact segmentation refinement, moving beyond traditional discriminative or dense representation methods.

Findings

Effective coarse-to-fine segmentation refinement demonstrated.

Outperforms traditional discriminative approaches.

Plug-and-play design validated through experiments.

Abstract

This paper considers the problem of utilizing a large-scale text-to-image diffusion model to tackle the challenging Inexact Segmentation (IS) task. Unlike traditional approaches that rely heavily on discriminative-model-based paradigms or dense visual representations derived from internal attention mechanisms, our method focuses on the intrinsic generative priors in Stable Diffusion~(SD). Specifically, we exploit the pattern discrepancies between original images and mask-conditional generated images to facilitate a coarse-to-fine segmentation refinement by establishing a semantic correspondence alignment and updating the foreground probability. Comprehensive quantitative and qualitative experiments validate the effectiveness and superiority of our plug-and-play design, underscoring the potential of leveraging generation discrepancies to model dense representations and encouraging further exploration of generative approaches for solving discriminative tasks.