SDMatte: Grafting Diffusion Models for Interactive Matting

TL;DR

SDMatte introduces a diffusion-based interactive matting approach that leverages text-driven and visual prompts, integrating spatial and opacity information with a masked self-attention mechanism for superior fine-detail extraction.

Contribution

The paper presents a novel diffusion-driven interactive matting model that transforms text prompts into visual prompts, integrating spatial and opacity cues with a masked self-attention mechanism.

Findings

Outperforms existing methods on multiple datasets

Effectively captures fine-grained edge details

Demonstrates robustness in interactive matting tasks

Abstract

Recent interactive matting methods have shown satisfactory performance in capturing the primary regions of objects, but they fall short in extracting fine-grained details in edge regions. Diffusion models trained on billions of image-text pairs, demonstrate exceptional capability in modeling highly complex data distributions and synthesizing realistic texture details, while exhibiting robust text-driven interaction capabilities, making them an attractive solution for interactive matting. To this end, we propose SDMatte, a diffusion-driven interactive matting model, with three key contributions. First, we exploit the powerful priors of diffusion models and transform the text-driven interaction capability into visual prompt-driven interaction capability to enable interactive matting. Second, we integrate coordinate embeddings of visual prompts and opacity embeddings of target objects into U-Net, enhancing SDMatte's sensitivity to spatial position information and opacity information. Third, we propose a masked self-attention mechanism that enables the model to focus on areas specified by visual prompts, leading to better performance. Extensive experiments on multiple datasets demonstrate the superior performance of our method, validating its effectiveness in interactive matting. Our code and model are available at https://github.com/vivoCameraResearch/SDMatte.