gen2seg: Generative Models Enable Generalizable Instance Segmentation

TL;DR

This paper demonstrates that generative models like Stable Diffusion and MAE can be finetuned for category-agnostic instance segmentation, showing strong zero-shot generalization to unseen object types and styles.

Contribution

The authors show that generative models can be repurposed for generalizable instance segmentation, outperforming existing promptable architectures and closely approaching supervised methods.

Findings

Models exhibit strong zero-shot generalization to unseen objects.

Generative models outperform existing promptable segmentation architectures.

Models closely approach supervised segmentation performance on unseen categories.

Abstract

By pretraining to synthesize coherent images from perturbed inputs, generative models inherently learn to understand object boundaries and scene compositions. How can we repurpose these generative representations for general-purpose perceptual organization? We finetune Stable Diffusion and MAE (encoder+decoder) for category-agnostic instance segmentation using our instance coloring loss exclusively on a narrow set of object types (indoor furnishings and cars). Surprisingly, our models exhibit strong zero-shot generalization, accurately segmenting objects of types and styles unseen in finetuning. This holds even for MAE, which is pretrained on unlabeled ImageNet-1K only. When evaluated on unseen object types and styles, our best-performing models closely approach the heavily supervised SAM, and outperform it when segmenting fine structures and ambiguous boundaries. In contrast, existing promptable segmentation architectures or discriminatively pretrained models fail to generalize. This suggests that generative models learn an inherent grouping mechanism that transfers across categories and domains, even without internet-scale pretraining. Please see our website for additional qualitative figures, code, and a demo.