Generalize Polyp Segmentation via Inpainting across Diverse Backgrounds and Pseudo-Mask Refinement

TL;DR

This paper introduces a novel inpainting-based data augmentation approach using pre-trained generative models to improve polyp segmentation across diverse backgrounds, achieving results comparable to fully supervised methods.

Contribution

The work develops a new inpainting and pseudo-mask refinement framework leveraging Stable Diffusion and ControlNet for enhanced polyp segmentation generalization.

Findings

Outperforms baseline inpainting methods qualitatively and quantitatively.

Data augmentation with our method improves segmentation on external datasets.

Achieves or surpasses fully supervised training benchmarks.

Abstract

Inpainting lesions within different normal backgrounds is a potential method of addressing the generalization problem, which is crucial for polyp segmentation models. However, seamlessly introducing polyps into complex endoscopic environments while simultaneously generating accurate pseudo-masks remains a challenge for current inpainting methods. To address these issues, we first leverage the pre-trained Stable Diffusion Inpaint and ControlNet, to introduce a robust generative model capable of inpainting polyps across different backgrounds. Secondly, we utilize the prior that synthetic polyps are confined to the inpainted region, to establish an inpainted region-guided pseudo-mask refinement network. We also propose a sample selection strategy that prioritizes well-aligned and hard synthetic cases for further model fine-tuning. Experiments demonstrate that our inpainting model outperformed baseline methods both qualitatively and quantitatively in inpainting quality. Moreover, our data augmentation strategy significantly enhances the performance of polyp segmentation models on external datasets, achieving or surpassing the level of fully supervised training benchmarks in that domain. Our code is available at https://github.com/497662892/PolypInpainter.