PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects

TL;DR

PartGLEE is a unified foundation model that recognizes and parses objects and their parts at any granularity, improving hierarchical understanding and perception in images for open-world scenarios.

Contribution

It introduces a hierarchical framework with Q-Former for part-level recognition, extending capabilities beyond previous models like GLEE.

Findings

Achieves state-of-the-art performance on part-level tasks

Obtains competitive results on object-level tasks

Enhances hierarchical modeling and detailed image comprehension

Abstract

We present PartGLEE, a part-level foundation model for locating and identifying both objects and parts in images. Through a unified framework, PartGLEE accomplishes detection, segmentation, and grounding of instances at any granularity in the open world scenario. Specifically, we propose a Q-Former to construct the hierarchical relationship between objects and parts, parsing every object into corresponding semantic parts. By incorporating a large amount of object-level data, the hierarchical relationships can be extended, enabling PartGLEE to recognize a rich variety of parts. We conduct comprehensive studies to validate the effectiveness of our method, PartGLEE achieves the state-of-the-art performance across various part-level tasks and obtain competitive results on object-level tasks. The proposed PartGLEE significantly enhances hierarchical modeling capabilities and part-level perception over our previous GLEE model. Further analysis indicates that the hierarchical cognitive ability of PartGLEE is able to facilitate a detailed comprehension in images for mLLMs. The model and code will be released at https://provencestar.github.io/PartGLEE-Vision/ .