Understanding Multi-Granularity for Open-Vocabulary Part Segmentation

TL;DR

This paper introduces PartCLIPSeg, a novel framework for open-vocabulary part segmentation that leverages generalized parts and object-level contexts to improve segmentation accuracy and understanding of part relationships in complex images.

Contribution

We propose PartCLIPSeg, a new method that enhances open-vocabulary part segmentation by integrating generalized parts, object-level contexts, and attention mechanisms to address boundary ambiguity and generalization issues.

Findings

Outperforms existing OVPS methods on multiple datasets

Achieves significant improvements in segmentation accuracy

Provides better understanding of part relationships

Abstract

Open-vocabulary part segmentation (OVPS) is an emerging research area focused on segmenting fine-grained entities using diverse and previously unseen vocabularies. Our study highlights the inherent complexities of part segmentation due to intricate boundaries and diverse granularity, reflecting the knowledge-based nature of part identification. To address these challenges, we propose PartCLIPSeg, a novel framework utilizing generalized parts and object-level contexts to mitigate the lack of generalization in fine-grained parts. PartCLIPSeg integrates competitive part relationships and attention control, alleviating ambiguous boundaries and underrepresented parts. Experimental results demonstrate that PartCLIPSeg outperforms existing state-of-the-art OVPS methods, offering refined segmentation and an advanced understanding of part relationships within images. Through extensive experiments, our model demonstrated a significant improvement over the state-of-the-art models on the Pascal-Part-116, ADE20K-Part-234, and PartImageNet datasets.