Explore In-Context Segmentation via Latent Diffusion Models

TL;DR

This paper introduces a novel approach to in-context segmentation using latent diffusion models, proposing new design strategies and a comprehensive benchmark, achieving competitive results compared to existing methods.

Contribution

It explores the application of latent diffusion models for in-context segmentation, introducing a two-stage masking strategy, pseudo-masking targets, and a new benchmark dataset.

Findings

Effective segmentation performance comparable to or better than existing models

New benchmark dataset covering image and video segmentation tasks

Validated design choices through extensive experiments

Abstract

In-context segmentation has drawn increasing attention with the advent of vision foundation models. Its goal is to segment objects using given reference images. Most existing approaches adopt metric learning or masked image modeling to build the correlation between visual prompts and input image queries. This work approaches the problem from a fresh perspective - unlocking the capability of the latent diffusion model (LDM) for in-context segmentation and investigating different design choices. Specifically, we examine the problem from three angles: instruction extraction, output alignment, and meta-architectures. We design a two-stage masking strategy to prevent interfering information from leaking into the instructions. In addition, we propose an augmented pseudo-masking target to ensure the model predicts without forgetting the original images. Moreover, we build a new and fair in-context segmentation benchmark that covers both image and video datasets. Experiments validate the effectiveness of our approach, demonstrating comparable or even stronger results than previous specialist or visual foundation models. We hope our work inspires others to rethink the unification of segmentation and generation.