GLaMM: Pixel Grounding Large Multimodal Model

TL;DR

GLaMM is a novel multimodal model capable of generating natural language responses with dense pixel-wise object grounding, supporting flexible interaction through visual and textual prompts, and is evaluated on a new large-scale grounded conversation dataset.

Contribution

Introduces GLaMM, the first model to generate grounded language responses with dense pixel-level object segmentation and flexible multimodal prompts, along with a new benchmark dataset.

Findings

GLaMM effectively grounds objects in natural language responses.

The model performs well on downstream tasks like segmentation and captioning.

Proposed dataset enables large-scale evaluation of grounded conversation generation.

Abstract

Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.