LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation

TL;DR

LangHOPS is a novel multimodal large language model framework that enables open-vocabulary hierarchical object-part segmentation by grounding hierarchies in language and leveraging rich reasoning capabilities, achieving state-of-the-art results.

Contribution

It introduces the first MLLM-based approach for open-vocabulary object-part segmentation, integrating language grounding into hierarchical parsing.

Findings

Surpasses previous methods by 5.5% AP on PartImageNet

Achieves 4.8% higher AP in cross-dataset scenarios

Attains 2.5% mIOU improvement on unseen parts in ADE20K

Abstract

We propose LangHOPS, the first Multimodal Large Language Model (MLLM) based framework for open-vocabulary object-part instance segmentation. Given an image, LangHOPS can jointly detect and segment hierarchical object and part instances from open-vocabulary candidate categories. Unlike prior approaches that rely on heuristic or learnable visual grouping, our approach grounds object-part hierarchies in language space. It integrates the MLLM into the object-part parsing pipeline to leverage its rich knowledge and reasoning capabilities, and link multi-granularity concepts within the hierarchies. We evaluate LangHOPS across multiple challenging scenarios, including in-domain and cross-dataset object-part instance segmentation, and zero-shot semantic segmentation. LangHOPS achieves state-of-the-art results, surpassing previous methods by 5.5% Average Precision (AP) (in-domain) and 4.8% (cross-dataset) on the PartImageNet dataset and by 2.5% mIOU on unseen object parts in ADE20K (zero-shot). Ablation studies further validate the effectiveness of the language-grounded hierarchy and MLLM driven part query refinement strategy. The code will be released here.