Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model

TL;DR

This paper introduces DiffPNG, a diffusion model-based framework for phrase-level visual grounding that leverages segmentation and refinement techniques to achieve zero-shot performance on the PNG dataset.

Contribution

The paper presents a novel diffusion model approach for phrase-level grounding, fully utilizing diffusion architecture for segmentation and zero-shot learning.

Findings

DiffPNG achieves strong zero-shot performance on PNG dataset.

The framework effectively decomposes localization, segmentation, and refinement.

Refinement with SAM improves segmentation quality.

Abstract

Recently, diffusion models have increasingly demonstrated their capabilities in vision understanding. By leveraging prompt-based learning to construct sentences, these models have shown proficiency in classification and visual grounding tasks. However, existing approaches primarily showcase their ability to perform sentence-level localization, leaving the potential for leveraging contextual information for phrase-level understanding largely unexplored. In this paper, we utilize Panoptic Narrative Grounding (PNG) as a proxy task to investigate this capability further. PNG aims to segment object instances mentioned by multiple noun phrases within a given narrative text. Specifically, we introduce the DiffPNG framework, a straightforward yet effective approach that fully capitalizes on the diffusion's architecture for segmentation by decomposing the process into a sequence of localization, segmentation, and refinement steps. The framework initially identifies anchor points using cross-attention mechanisms and subsequently performs segmentation with self-attention to achieve zero-shot PNG. Moreover, we introduce a refinement module based on SAM to enhance the quality of the segmentation masks. Our extensive experiments on the PNG dataset demonstrate that DiffPNG achieves strong performance in the zero-shot PNG task setting, conclusively proving the diffusion model's capability for context-aware, phrase-level understanding. Source code is available at \url{https://github.com/nini0919/DiffPNG}.