Amodal Instance Segmentation with Diffusion Shape Prior Estimation

TL;DR

This paper introduces AISDiff, a novel amodal instance segmentation method that leverages diffusion models for shape prior estimation, improving segmentation accuracy especially in occluded scenarios.

Contribution

We propose AISDiff with a Diffusion Shape Prior Estimation module that uses pretrained diffusion models to enhance amodal segmentation accuracy.

Findings

Outperforms existing methods on multiple AIS benchmarks.

Effectively models occlusions and predicts complete object shapes.

Utilizes pretrained diffusion models for rich shape prior extraction.

Abstract

Amodal Instance Segmentation (AIS) presents an intriguing challenge, including the segmentation prediction of both visible and occluded parts of objects within images. Previous methods have often relied on shape prior information gleaned from training data to enhance amodal segmentation. However, these approaches are susceptible to overfitting and disregard object category details. Recent advancements highlight the potential of conditioned diffusion models, pretrained on extensive datasets, to generate images from latent space. Drawing inspiration from this, we propose AISDiff with a Diffusion Shape Prior Estimation (DiffSP) module. AISDiff begins with the prediction of the visible segmentation mask and object category, alongside occlusion-aware processing through the prediction of occluding masks. Subsequently, these elements are inputted into our DiffSP module to infer the shape prior of the object. DiffSP utilizes conditioned diffusion models pretrained on extensive datasets to extract rich visual features for shape prior estimation. Additionally, we introduce the Shape Prior Amodal Predictor, which utilizes attention-based feature maps from the shape prior to refine amodal segmentation. Experiments across various AIS benchmarks demonstrate the effectiveness of our AISDiff.