SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process

TL;DR

SegRefiner is a novel, model-agnostic segmentation refinement method that uses discrete diffusion processes to improve mask quality across various segmentation tasks, outperforming previous methods.

Contribution

Introduces SegRefiner, a diffusion-based, model-agnostic approach for segmentation mask refinement, capable of enhancing boundary and detail accuracy across multiple segmentation types.

Findings

Consistently improves segmentation and boundary metrics.

Outperforms previous model-agnostic refinement methods.

Effectively captures fine details in high-resolution images.

Abstract

In this paper, we explore a principal way to enhance the quality of object masks produced by different segmentation models. We propose a model-agnostic solution called SegRefiner, which offers a novel perspective on this problem by interpreting segmentation refinement as a data generation process. As a result, the refinement process can be smoothly implemented through a series of denoising diffusion steps. Specifically, SegRefiner takes coarse masks as inputs and refines them using a discrete diffusion process. By predicting the label and corresponding states-transition probabilities for each pixel, SegRefiner progressively refines the noisy masks in a conditional denoising manner. To assess the effectiveness of SegRefiner, we conduct comprehensive experiments on various segmentation tasks, including semantic segmentation, instance segmentation, and dichotomous image segmentation. The results demonstrate the superiority of our SegRefiner from multiple aspects. Firstly, it consistently improves both the segmentation metrics and boundary metrics across different types of coarse masks. Secondly, it outperforms previous model-agnostic refinement methods by a significant margin. Lastly, it exhibits a strong capability to capture extremely fine details when refining high-resolution images. The source code and trained models are available at https://github.com/MengyuWang826/SegRefiner.