OV-PARTS: Towards Open-Vocabulary Part Segmentation

TL;DR

This paper introduces OV-PARTS, a new benchmark for open-vocabulary part segmentation, addressing challenges like intricate boundaries, open granularity, and limited annotations, to advance understanding of model capabilities in recognizing object parts.

Contribution

It proposes the OV-PARTS benchmark with refined datasets and tasks, and analyzes adaptation of existing object-level OVSS methods for part segmentation.

Findings

Benchmark reveals challenges in open-vocabulary part segmentation.

Analysis shows existing models struggle with fine-grained part recognition.

Provides insights for future research in foundational model adaptation.

Abstract

Segmenting and recognizing diverse object parts is a crucial ability in applications spanning various computer vision and robotic tasks. While significant progress has been made in object-level Open-Vocabulary Semantic Segmentation (OVSS), i.e., segmenting objects with arbitrary text, the corresponding part-level research poses additional challenges. Firstly, part segmentation inherently involves intricate boundaries, while limited annotated data compounds the challenge. Secondly, part segmentation introduces an open granularity challenge due to the diverse and often ambiguous definitions of parts in the open world. Furthermore, the large-scale vision and language models, which play a key role in the open vocabulary setting, struggle to recognize parts as effectively as objects. To comprehensively investigate and tackle these challenges, we propose an Open-Vocabulary Part Segmentation (OV-PARTS) benchmark. OV-PARTS includes refined versions of two publicly available datasets: Pascal-Part-116 and ADE20K-Part-234. And it covers three specific tasks: Generalized Zero-Shot Part Segmentation, Cross-Dataset Part Segmentation, and Few-Shot Part Segmentation, providing insights into analogical reasoning, open granularity and few-shot adapting abilities of models. Moreover, we analyze and adapt two prevailing paradigms of existing object-level OVSS methods for OV-PARTS. Extensive experimental analysis is conducted to inspire future research in leveraging foundational models for OV-PARTS. The code and dataset are available at https://github.com/OpenRobotLab/OV_PARTS.