ZIM: Zero-Shot Image Matting for Anything

TL;DR

ZIM introduces a zero-shot image matting model that leverages a novel dataset and architecture to produce fine-grained masks, enhancing zero-shot segmentation and enabling diverse applications like inpainting and 3D modeling.

Contribution

The paper presents a new zero-shot image matting approach with a label converter, a hierarchical pixel decoder, and a prompt-aware attention mechanism, improving fine-grained mask generation without manual annotations.

Findings

ZIM outperforms existing methods in fine-grained mask generation.

ZIM demonstrates strong zero-shot generalization across tasks.

The approach enables applications like image inpainting and 3D NeRF.

Abstract

The recent segmentation foundation model, Segment Anything Model (SAM), exhibits strong zero-shot segmentation capabilities, but it falls short in generating fine-grained precise masks. To address this limitation, we propose a novel zero-shot image matting model, called ZIM, with two key contributions: First, we develop a label converter that transforms segmentation labels into detailed matte labels, constructing the new SA1B-Matte dataset without costly manual annotations. Training SAM with this dataset enables it to generate precise matte masks while maintaining its zero-shot capability. Second, we design the zero-shot matting model equipped with a hierarchical pixel decoder to enhance mask representation, along with a prompt-aware masked attention mechanism to improve performance by enabling the model to focus on regions specified by visual prompts. We evaluate ZIM using the newly introduced MicroMat-3K test set, which contains high-quality micro-level matte labels. Experimental results show that ZIM outperforms existing methods in fine-grained mask generation and zero-shot generalization. Furthermore, we demonstrate the versatility of ZIM in various downstream tasks requiring precise masks, such as image inpainting and 3D NeRF. Our contributions provide a robust foundation for advancing zero-shot matting and its downstream applications across a wide range of computer vision tasks. The code is available at https://github.com/naver-ai/ZIM.