Referring Image Matting

TL;DR

This paper introduces Referring Image Matting (RIM), a new task that extracts specific object mattes based on natural language descriptions, supported by a large dataset and a novel baseline method.

Contribution

The paper presents a new RIM task, a large-scale dataset RefMatte, and a baseline method CLIPMat, advancing image matting with language-guided object extraction.

Findings

CLIPMat outperforms existing methods on RefMatte.

RefMatte dataset contains 230 categories and 47,500 images.

CLIPMat effectively handles complex natural language descriptions.

Abstract

Different from conventional image matting, which either requires user-defined scribbles/trimap to extract a specific foreground object or directly extracts all the foreground objects in the image indiscriminately, we introduce a new task named Referring Image Matting (RIM) in this paper, which aims to extract the meticulous alpha matte of the specific object that best matches the given natural language description, thus enabling a more natural and simpler instruction for image matting. First, we establish a large-scale challenging dataset RefMatte by designing a comprehensive image composition and expression generation engine to automatically produce high-quality images along with diverse text attributes based on public datasets. RefMatte consists of 230 object categories, 47,500 images, 118,749 expression-region entities, and 474,996 expressions. Additionally, we construct a real-world test set with 100 high-resolution natural images and manually annotate complex phrases to evaluate the out-of-domain generalization abilities of RIM methods. Furthermore, we present a novel baseline method CLIPMat for RIM, including a context-embedded prompt, a text-driven semantic pop-up, and a multi-level details extractor. Extensive experiments on RefMatte in both keyword and expression settings validate the superiority of CLIPMat over representative methods. We hope this work could provide novel insights into image matting and encourage more follow-up studies. The dataset, code and models are available at https://github.com/JizhiziLi/RIM.